Aircraft door camera system for docking alignment monitoring

ABSTRACT

A camera with a field of view toward an external environment of an aircraft is disposed within an aircraft door such that a ground surface is within the field of view of the camera during taxiing of the aircraft. A display device is disposed within an interior of the aircraft. A processor is operatively coupled to the camera and to the display device. The processor analyzes image data captured by the camera for docking guidance by identifying, within the captured image data, a region on the ground surface corresponding to an alignment fiducial indicating a parking location for the aircraft, determining, based on the region of the captured image data corresponding to the alignment fiducial indicating the parking location, a relative location of the aircraft with respect to the alignment fiducial, and outputting an indication of the relative location of the aircraft to the alignment fiducial.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.63/231,840 filed Aug. 11, 2021 for “AIRCRAFT DOOR CAMERA SYSTEM FORMONITORING AN EXTERNAL ENVIRONMENT OF THE AIRCRAFT” by J. Pesik and J.Boer.

This application is related to U.S. application Ser. No. ______ filedAug. 11, 2022 for “AIRCRAFT DOOR CAMERA SYSTEM FOR EVACUATION SLIDEDEPLOYMENT MONITORING” by J. Pesik and J. Boer, U.S. application Ser.No. ______ filed Aug. 11, 2022 for “AIRCRAFT DOOR CAMERA SYSTEM FOR WINGMONITORING” by J. Pesik and J. Boer, U.S. application Ser. No. ______filed Aug. 11, 2022 for “AIRCRAFT DOOR CAMERA SYSTEM FOR ENGINE INLETMONITORING” by J. Pesik and J. Boer, U.S. application Ser. No. ______filed Aug. 11, 2022 for “AIRCRAFT DOOR CAMERA SYSTEM FOR LANDING GEARMONITORING” by J. Pesik and J. Boer, and U.S. application Ser. No.______ filed Aug. 11, 2022 for “AIRCRAFT DOOR CAMERA SYSTEM FOR JETBRIDGE ALIGNMENT MONITORING” by J. Pesik and J. Boer.

The specifications of each of these applications are incorporated hereinby reference in their entirety.

BACKGROUND

The present disclosure relates generally to aircraft monitoring systems,and more particularly to aircraft monitoring systems including camerasdisposed within aircraft doors for generating views external to theaircraft.

Modern aircraft are typically outfitted with multiple entry orevacuation doors. Passengers may pass through these doors duringboarding or deboarding operations or during emergency evacuations. Suchaircraft doors are often constructed with a window providing the crew aview to an external environment of the aircraft. Window features, suchas size and location within the door, are constrained by the doorarchitecture. Further, these windows must be designed to withstandenvironments about the aircraft such as high speeds, cold temperatures,low external pressures, and pressurized aircraft cabins. As a result,such windows typically have significant supporting structures thatresult in limited window size and viewing angles.

The view provided by these windows may be used in a variety of phases offlight. For instance, the crew may utilize the aircraft door windows todetermine whether an evacuation slide deployment path is unobstructedand safe during emergency operations. Due at least in part to thetypically limited size of the windows, obtaining a full understanding ofthe scene outside the door often requires a crew member to move his orher head and eyes side to side and up and down, from edge to edge of thewindow. Additional movement necessitated by the limited size of thewindow may detract from the crew's duties and, in emergency situations,may result in delayed execution of safe evacuation procedures.

SUMMARY

In one example, a system for monitoring an external environment of theaircraft includes an aircraft door, a camera, a display device, and aprocessor. The camera has a field of view toward the externalenvironment of the aircraft and is disposed within an aircraft door suchthat a ground surface is within the field of view of the camera duringtaxiing of the aircraft. The display device is disposed within aninterior of the aircraft. The processor is operatively coupled to thecamera and display device. The processor receives image data captured bythe camera that is representative of the external environment of theaircraft and outputs the captured image data for display at the displaydevice. The processor analyzes the captured image data for dockingguidance by: identifying, within the captured image data, a region onthe ground surface corresponding to an alignment fiducial indicating aparking location for the aircraft, determining, based on the region ofthe captured image data corresponding to the alignment fiducialindicating the parking location, a relative location of the aircraftwith respect to the alignment fiducial, and outputting an indication ofthe relative location of the aircraft with respect to the alignmentfiducial.

In another example, a method of monitoring an external environment of anaircraft includes receiving, with a processor, image data captured by acamera disposed within an aircraft door of the aircraft such that aground surface is within a field of view of the camera during taxiing ofthe aircraft. The processor analyzes the captured image data for dockingguidance by: identifying, within the captured image data, a region onthe ground surface corresponding to an alignment fiducial indicating aparking location for the aircraft, determining, based on the region ofthe captured image data corresponding to the alignment fiducialindicating the parking location, a relative location of the aircraftwith respect to the alignment fiducial, and outputting an indication ofthe relative location of the aircraft with respect to the alignmentfiducial. The captured image data is output for display at a displaydevice disposed within an interior of the aircraft.

In another example, a system of monitoring an external environment of anaircraft includes a plurality of aircraft doors, a plurality of cameras,a display device, and a processor. At least one of the plurality ofcameras are disposed within one of the aircraft doors and each of theplurality of cameras have a field of view that is unique among theplurality of cameras. A ground surface is within the field of view of atleast one camera during taxiing of the aircraft. The display device isdisposed within an interior of the aircraft. The processor isoperatively coupled to the camera and display device to: receive, fromeach respective camera of the plurality of cameras, image data capturedby the respective camera that is representative of the externalenvironment of the aircraft within the field of view of the respectivecamera, aggregate the captured image data received from each camera ofthe plurality of cameras to produce aggregated image data representativeof the external environment of the aircraft, wherein image data fromoverlapping fields of view of the plurality of cameras is presented onlyonce in the aggregated image data, and output the aggregated image datafor display at the display device. The processor analyzes the aggregatedimage data for docking guidance by: identifying, within the aggregatedimage data, a region on the ground surface corresponding to an alignmentfiducial indicating a parking location for the aircraft, determining,based on the region of the aggregated image data corresponding to thealignment fiducial indicating the parking location, a relative locationof the aircraft with respect to the alignment fiducial, and outputtingan indication of the relative location of the aircraft with respect tothe alignment fiducial.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of an aircraft including doors havingcamera systems with fields of view toward an exterior of the aircraft.

FIG. 2 is a partial cross-sectional view of the camera disposed withinan aircraft door and operatively coupled with a display device.

FIG. 3 illustrates a representation of the camera disposed within theaircraft door below a vertical midpoint of the aircraft door.

FIG. 4A is a perspective view of the camera assembly including anelectronics housing and a mounting gasket.

FIG. 4B is a perspective view of the camera assembly showing a mountingflange in relation to a viewing window of the camera.

FIG. 5 is a block diagram illustrating components of the camera systemin communication with the display and aircraft avionics equipment.

FIG. 6 is a top-down view of an aircraft including multiple camerasystems disposed within doors of the aircraft and having overlappingfields of view.

FIG. 7A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning of anticipated collision based on an identified regionwithin the captured image data that corresponds to an edge of a wing ofthe aircraft.

FIG. 7B is a schematic depiction of the aircraft of FIG. 1 , includingfields of view of the cameras of FIG. 1 , damage to a wing of theaircraft, and an object separate from the aircraft.

FIG. 8 is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning in response to determining that a region within thecaptured image data that corresponds to a leading edge of a wing of theaircraft does not conform to baseline image data.

FIG. 9A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning corresponding to proximity of ground personnel to anengine inlet.

FIG. 9B is a schematic depiction of the aircraft of FIG. 1 , includingfields of view of the cameras of FIG. 1 , and ground personnel about theaircraft.

FIG. 10A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning corresponding to ingestion of a foreign object to anengine inlet of an engine of the aircraft.

FIG. 10B is a schematic depiction of the aircraft of FIG. 1 , includingfields of view of the cameras of FIG. 1 , and an object separate fromthe aircraft.

FIG. 11A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning associated with an engine inlet in response todetermining that an identified region within the capture image data thatcorresponds to the engine inlet does not conform to baseline image data.

FIG. 11B is a schematic depiction of the aircraft of FIG. 1 , includingfields of view of the cameras of FIG. 1 , and damage to an engine inletof the aircraft.

FIG. 12A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce an output that indicates whether an identified region within thecaptured image data that corresponds to a wheel of main landing gear ofthe aircraft includes image data corresponding to a chock block.

FIG. 12B is a schematic depiction of the aircraft of FIG. 1 , includingfields of view of the cameras of FIG. 1 , and chock blocks present aboutlanding gear of the aircraft.

FIG. 12C is a schematic depiction of the aircraft of FIG. 1 , includingfields of view of the cameras of FIG. 1 , when landing gear of theaircraft is not fully extended.

FIG. 13A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door tooutput an indication of an alignment state between the aircraft door anda cabin of a jet bridge.

FIG. 13B is a front perspective view of the aircraft of FIG. 1 ,including a field of view of a camera of FIG. 1 , adjacent to a jetbridge.

FIG. 13C-13D are perspective views of the jet bridge of FIG. 13B.

FIG. 13E is a schematic depiction of the display device of FIG. 5outputting a graphical overlay of alignment features.

FIG. 14A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door tooutput an indication of a relative location of the aircraft to analignment fiducial indicating a parking location for the aircraft.

FIG. 14B is a schematic depiction of the aircraft of FIG. 1 , includingfields of view of the cameras of FIG. 1 , relative to docking fiducialson the ground proximate to the aircraft.

FIG. 15 is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce an incursion warning output that indicates an anticipatedcollision between the aircraft and an object at a runway intersection.

FIG. 16A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning associated with evacuation slide deployment inresponse to determining that an identified region within the capturedimage data that corresponds to an evacuation slide deployment path doesnot conform to baseline image data.

FIG. 16B is a schematic depiction of the aircraft of FIG. 1 , includingfields of view of the cameras of FIG. 1 , and an evacuation slidedeployment path.

FIG. 16C is a schematic depiction of the evacuation slide deploymentpath of FIG. 16B against various terrain heights relative to theevacuation slide deployment path.

FIG. 17 is a flow chart illustrating example operations for aggregatingcaptured image data received from each of a plurality of camerasdisposed in a plurality of aircraft doors and having a unique field ofview to produce aggregated image data representative of the externalenvironment of the aircraft.

DETAILED DESCRIPTION

As described herein, an example aircraft monitoring system includes acamera disposed within an aircraft door that provides image data of afield of view of an external environment of the aircraft. The camerasystem can, in some examples, replace the window of the aircraft door,such that the aircraft door is windowless, thereby decreasing the weightof the aircraft door by eliminating the need for robust supportstructures that are typically utilized to enable such windows towithstand the operational environment of the aircraft. Moreover,placement of the camera at the skin of the aircraft can provide widerviewing angles than would otherwise be achievable through a physicalwindow of the aircraft door (due in part to viewing angle limitationsimposed by the thickness of the aircraft door). In some examples,multiple camera systems can be used to provide multiple (e.g., two,three, or more) independent views of the physical conditions of theenvironment about the aircraft, each camera providing a different,independent field of view.

According to techniques of this disclosure, the image data captured bythe camera systems can be further analyzed via image processingoperations to provide alerts, automatic guidance, or other outputs thatcan reduce aircraft crew workload, thereby increasing crew efficiency aswell as the safety of the aircraft crew and passengers during aircraftoperation. For instance, as is further described below, captured imagedata from the camera system (or systems) disposed within the aircraftdoor (or doors) can be analyzed to provide alerts and/or guidancerelated to wing edge collision avoidance, wing leading edge deformation(e.g., due to ice accretion or foreign object damage), safety-clearancebetween ground personnel and an engine inlet, engine inlet foreignobject ingestion, engine inlet damage visualization, presence of a chockblock at, e.g., a main landing gear of the aircraft, jet bridgealignment during docking operations, surface marking alignment,incursion warning, and the aggregation of image data from multiplecameras to provide increased situational awareness of the externalenvironment of the aircraft. Accordingly, camera systems implementingtechniques of this disclosure can effectively replace windows within theaircraft doors, thereby decreasing the weight and cost associated withsupport structures within the aircraft doors that enable the windows towithstand the operational environment of the aircraft. Moreover, imagedata captured by the one or more cameras can be analyzed to providealerts, guidance, or other outputs that can reduce crew workload andincrease crew efficiency and safety of the passengers and aircraftflight crew.

FIG. 1 is a perspective view of aircraft 10 with aircraft skin 12,including doors 14 a, 14 b, and 14 c having cameras 16 a, 16 b, and 16 cdisposed therein. As illustrated in FIG. 1 , cameras 16 a, 16 b, and 16c can be disposed within aircraft doors 14 a, 14 b, and 14 c,respectively. While the example of FIG. 1 illustrates three camerasdisposed within three aircraft doors, it should be understood that inother examples, other numbers of camera systems can be incorporated inaircraft doors, such as one camera (e.g., within a single door), twocameras (e.g., each within a separate door), or three or more camerasdisposed within three or more aircraft doors. In certain examples, morethan one camera can be disposed within a single door. In yet otherexamples, not every door need have a camera disposed therein, meaningthat one or more aircraft doors incorporates a camera and one or moreaircraft doors does not incorporate a camera.

As in the example of FIG. 1 , cameras 16 a, 16 b, and 16 c can bedisposed within aircraft doors 14 a, 14 b, and 14 c, respectively, suchthat cameras 16 a, 16 b, and 16 c are flush with aircraft skin 12 tomaintain aerodynamic efficiency and to reduce drag. For instance, as isfurther described below, cameras 16 a, 16 b, and 16 c can be disposedwithin aircraft doors 14 a, 14 b, and 14 c, such that an outermostsurface of cameras 16 a, 16 b, and 16 c (e.g., a lens of the camera orother outer-most surface of the camera) is flush with (i.e., even with)aircraft skin 12. In other examples, cameras 16 a, 16 b, and 16 c neednot be disposed flush with aircraft skin 12.

As illustrated in FIG. 1 , cameras 16 a, 16 b, and 16 c are disposedsuch that a field of view of cameras 16 a, 16 b, and 16 c is orientedtoward an exterior of aircraft 10. Each of cameras 16 a, 16 b, 16 c isoperatively (e.g. communicatively and/or electrically) connected to aprocessor (not illustrated) and to a display device (not illustrated) toa provide a visual representation of the field of view of the respectivecamera. In some examples, each of cameras 16 a, 16 b, and 16 c can beoperatively connected to a separate processor and display device, thoughcameras 16 a, 16 b, and 16 c need not be connected to separateprocessors and display devices in all examples. For instance, any two ormore of cameras 16 a, 16 b, and 16 c can be operatively connected to asame processor and/or a same display device.

Cameras 16 a, 16 b, and 16 c are configured to capture image data from afield of view external to the aircraft. Any one or more of cameras 16 a,16 b, and 16 c can be visible light spectrum cameras, infrared spectrumcameras, or other types of cameras capable of capturing image datawithin a field of view external to the aircraft. In some examples, anyone or more of cameras 16 a, 16 b, and 16 c can include or beaccompanied by a light source, such as a light emitting diode (LED) orother light source to illuminate at least a portion of the field of viewof the respective camera to improve visibility and the ability of thecamera to capture image data in low-light scenarios. The processor (notillustrated in FIG. 1 ) receives the captured image data, processes thecaptured image data, and communicates the captured image data to thedisplay device. The display device (not illustrated in FIG. 1 ) displaysthe image data received from the processor. As such, cameras 16 a, 16 b,and 16 c provide a visual representation to the flight crew and othersof the external environment of the aircraft for use in multiple phasesof flight.

In certain examples, cameras 16 a, 16 b, and 16 c can be configured tobe installed in the volume previously occupied by a window of the door,thereby serving as a replacement for the window. In such examples, anyone or more of aircraft doors 14 a, 14 b, and 14 c can be windowless,meaning that the respective door does not include a window, but ratherincludes a respective one of cameras 16 a, 16 b, and 16 c that areoperatively coupled to a display device to provide the field of view ofthe external environment of aircraft 10. In such examples, as is furtherdescribed below, cameras 16 a, 16 b, and 16 c can provide a field ofview that is greater than would be otherwise achievable via acorresponding window. That is, cameras 16 a, 16 b, and 16 c, disposed ator near aircraft skin 12, can provide a greater viewing angle than wouldotherwise be achievable through a window of the door due to thelimitations imposed on the viewing angle through the window by thethickness of the door and the corresponding support structures for thewindow that typically limit the size of the window. Moreover, image datacaptured by cameras 16 a, 16 b, and 16 c can be analyzed via imageprocessing operations to provide alerts, guidance, or other outputs thatcan reduce crew workload and increase safety of operations.

FIG. 2 is a partial cross-sectional view of camera 16 a disposed withinaircraft door 14 a with a field of view toward an exterior of aircraft10. Though the example of FIG. 2 is described below within the contextof camera 16 a for purposes of clarity and ease of discussion, it shouldbe understood that the techniques described below with respect to FIG. 2are applicable to any one or more of cameras 16 a, 16 b, and 16 c.

As illustrated in FIG. 2 , camera 16 a includes lens 18. Camera 16 a isoperatively (e.g. communicatively and/or electrically) connected withdisplay device 20. Camera 16 a can be operatively connected to displaydevice 20 via wired or wireless connection, or both. As is furtherdescribed below, camera 16 a is also operatively connected with aprocessor (not illustrated in FIG. 1 ) that is configured to receiveimage data captured by camera 16 a and to provide a representation(e.g., a graphical representation) of the captured image data to displaydevice 20 to provide a visual representation of the external environmentof aircraft 10.

In the example of FIG. 2 , camera 16 a is disposed within aircraft door14 a flush with aircraft skin 12, such that an outermost portion ofcamera 16 a (i.e., an outermost surface of lens 18 and other housing andmounting structures of camera 16 a) is flush with (i.e., even with)aircraft skin 12. In other examples, any portion of camera 16 a canprotrude from aircraft skin 12 into an oncoming airflow about aircraftskin 12. In yet other examples, any one or more portions of camera 16 acan be recessed within aircraft skin 12.

As illustrated in FIG. 2 , the field of view of camera 16 a is orientedtoward an exterior of aircraft 10 through lens 18. Image capturingelectronics (not illustrated) of camera 16 a are positioned withincamera 16 a to provide the field of view having angle a. Captured imagedata within the field of view is provided to display device 20 thatgraphically presents a visual depiction of the exterior of aircraft 10.

Display device 20 can be a liquid crystal display (LCD), an organiclight emitting diode (OLED) display, or other type of display devicecapable of providing graphical output of the image data captured bycamera 16 a to a user. As illustrated in FIG. 2 , display device 20 canbe mounted to the inside of door 14 a opposite camera 16 a. In otherexamples, display device 20 can be mounted to a wall surface withinaircraft 10, such as a wall surface adjacent to camera 16 a or in otherareas of an interior of aircraft 10. In yet other examples, displaydevice 20 can be a mobile display device, such as a tablet computer orother mobile display device, that can output the image data captured bycamera 16 a while positioned at any location throughout aircraft 10. Incertain examples, display device 20 can be mounted in the cockpit of theaircraft 10 or can be part of an existing cockpit display system, suchas an electronic flight instrument system (EFIS).

Accordingly, camera 16 a that is operatively connected to display device20 can provide a graphical representation of an exterior of aircraft 10to a flight crew or other user. The combination of camera 16 a anddisplay device 20 can be utilized to effectively replace a window in theaircraft door, such that the aircraft door can be constructed to bewindowless and without the supporting structure that is commonlyassociated with windows in aircraft doors, and without compromising theutility of a door that includes a window for viewing the exterior of theaircraft. Moreover, as is further described below, image data capturedby camera 16 a can be analyzed via image processing techniques toprovide alerts or other guidance to aircraft crew to increase efficiencyof the crew and safety of aircraft operation.

FIG. 3 illustrates a representation of camera 16 a disposed withinaircraft door 14 a at or below vertical center of curvature 22 ofaircraft door 14 a. While the example of FIG. 3 is described belowwithin the context of camera 16 a and door 14 a for purposes of clarityand ease of discussion, it should be understood that the techniquesdescribed below with respect to FIG. 3 are applicable to any one or moreof cameras 16 a, 16 b, and 16 c disposed within any of doors 14 a, 14 b,and 14 c. Moreover, while illustrated and described in the example ofFIG. 3 as being mounted below vertical center of curvature 22 ofaircraft door 14 a, camera 16 a can be mounted at other locations withinaircraft door 14 a, such as in a volume configured to contain a windowand/or associated support structure of aircraft door 14 a, or otherlocations within aircraft door 14 a.

Vertical center of curvature 22, as illustrated in FIG. 3 , cancorrespond to a location of door 14 a at which a line tangential to theaircraft skin at the outer surface of door 14 a is vertical (e.g., amidpoint of a vertical height of the fuselage of an aircraft), belowwhich an orthogonal vector to the fuselage is oriented in a downwarddirection. Camera 14 a, disposed below vertical center of curvature 22has a field of view angled toward a surface of the ground. As such,camera 16 a when disposed below vertical center of curvature 22 canprovide views of features below camera 16 a, such as a main landinggear, an engine inlet, a leading edge of a wing, an aft edge of thewing, surface markings for docking alignment, jet bridge features, orother features within the field of view of camera 16 a.

As is further described below, a processor (not illustrated) can performanalysis based in part on the features captured in the field of view ofcamera 16 a, thereby providing information relevant to operationalcontrol of the aircraft, such as alerts, guidance of other operationalinformation.

FIG. 4A is a perspective view of camera 16 a showing electronics housing24, input/output connector 26, power connector 28, and mounting gasket30 on a back side of mounting flange 32. FIG. 4B is a perspective viewof camera 16 a showing a front side of mounting flange 32 includingmounting bores that are utilized for mounting camera 16 a to an externalsurface of an aircraft. The examples of FIGS. 4A and 4B are describedbelow together for purposes of clarity and ease of discussion. Moreover,it should be understood that while the examples of FIGS. 4A and 4B aredescribed below with respect to camera 16 a, the examples of FIGS. 4Aand 4B are applicable to any of cameras 16 a, 16 b, and 16 c.

Electronics housing 24 is configured to enclose electrical and othercomponents of camera 16 a, such as one or more processors, memory, lenscomponents, image sensor components, or other components of camera 16 a.Electronics housing 24, as illustrated in FIGS. 4A and 4B, can enclose(e.g., all) components of camera 16 a, such that camera 16 a can beconsidered a line replaceable unit (LRU) in some examples. Powerconnector 28 is electrically connected to components within the interiorof electronics housing 24 to provide electrical power to the componentsduring operation. Input/output connector 26 is connected to electricalcomponents within the interior of electronics housing 24 forcommunication between camera 16 a and components of the aircraft thatare remote from camera 16 a, such as aircraft avionics components, adisplay device (or devices), or other components. Camera 16 a can beconfigured to communicate over an aircraft data bus via input/outputconnector 26, such as via the Aeronautical Radio, Incorporated (ARINC)429 interface, a Controller Area Network (CAN) bus network, or othercommunication network. Though the example of FIG. 4A illustrates powerconnector 28 and input/output connector 26 as separate connectors, insome examples, power connector 28 and input/output connector 26 can becombined into a single connector that provides both electrical power andcommunication capabilities for camera 16 a.

Mounting flange 32 is utilized to mount camera 16 a to the aircraft.Mounting bores within mounting flange 32 can be utilized for securingmounting flange 32 (and therefore camera 16 a) to the aircraft. As inthe example of FIGS. 4A and 4B, mounting flange 32 can be utilized tosecure camera 16 a to the aircraft from the outside of the aircraft(i.e., rather than mounting from within the aircraft door) for ease ofinstallation, maintenance, and replacement. Mounting gasket 30, adhered(e.g., adhesively adhered or otherwise adhered) to a back side ofmounting flange 32 and configured to make contact with the aircraft caneffectively seal the connection between mounting flange 32 and theaircraft structure to prevent ingress of water, particulates, or othercontaminants.

Mounting flange 32 can be configured to be mounted flush (i.e., even)with an outer skin of the aircraft to reduce drag and maintainaerodynamic efficiency of the aircraft. For instance, mounting flange 32can be configured to be installed within a recess produced on the outerskin of the aircraft at an exterior of the aircraft door, such that anoutermost surface of mounting flange 32 is flush with the aircraft skin.In other examples, mounting flange 32 can protrude from the aircraftskin into the airflow about the exterior of the aircraft.

Camera 16 a can therefore be disposed within an aircraft door to capturea field of view of the external environment and external components ofthe aircraft. Camera 16 a, in some examples, can be mounted flush withthe aircraft skin to maintain aerodynamic efficiency of the aircraftwhile capturing a field of view that is greater than would otherwise beachievable via a window in the door of the aircraft.

FIG. 5 is a block diagram illustrating components of camera 16 a incommunication with display device 20 and aircraft avionics equipment 34.While described below with respect to camera 16 a for purposes ofclarity and ease of discussion, it should be understood that thetechniques described below with respect to FIG. 5 are applicable to anyone or more of cameras 16 a, 16 b, and 16 c.

As illustrated in FIG. 5 , camera 16 a includes processor 36 andcomputer-readable memory 37. Camera 16 a, processor 36, andcomputer-readable memory 37 can be disposed within an electronicshousing, such as electronics housing 24 (FIGS. 4A and 4B). Camera 16 ais electrically and/or communicatively coupled with processor 36. Insome examples, processor 36 and/or computer-readable memory 37 can beconsidered part of camera 16 a (e.g., integral to camera 16 a). In otherexamples, any one or more of processor 36 and computer-readable memory37 can be separate from and electrically and/or communicatively coupledwith camera 16 a. Alerting module 39 can be a component of processor 36(e.g., integral to processor 36) or aircraft avionics equipment 24, orcan be a separate hardware or software component within aircraft 10.

Processor 36, in some examples, is configured to implement functionalityand/or process instructions for execution during operation of camera 16a. For instance, processor 36 can be capable of processing instructionsstored in computer-readable memory 37. Examples of processor 36 caninclude any one or more of a microprocessor, a controller, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a field-programmable gate array (FPGA), or other equivalentdiscrete or integrated logic circuitry.

Computer-readable memory 37 can be configured to store information foruse by processor 36 or other components of camera 16 a during operationof camera 16 a. Computer-readable memory 37, in certain examples, caninclude a non-transitory medium. The term “non-transitory” can indicatethat the storage medium is not embodied in a carrier wave or apropagated signal. In some examples, a non-transitory storage medium canstore data that can, over time, change (e.g., in RAM or cache).Computer-readable memory can include volatile and/or non-volatile memoryelements. Examples of volatile memory elements can include random accessmemories (RAM), dynamic random access memories (DRAM), static randomaccess memories (SRAM), and other forms of volatile memories. Examplesof non-volatile memory elements can include magnetic hard discs, opticaldiscs, flash memories, or forms of electrically programmable memories(EPROM) or electrically erasable and programmable (EEPROM) memories.

Processor 36, as illustrated in FIG. 5 , can be operatively connected(e.g., electrically and/or communicatively connected) with camera 16 a,computer-readable memory 37, display device 20, and aircraft avionics34. Processor 36, in certain examples, can include and/or be furtherconnected to communication components, graphics processing components,or other electrical components for facilitating communication betweencomponents (or via an aircraft data bus) and for processing image datacaptured by camera 16 a for image processing operations and display atdisplay device 20.

In operation, camera 16 a captures image data within a field of view ofcamera 16 a that is oriented toward an exterior of the aircraft. Imagedata captured by camera 16 a is processed by processor 36 and output todisplay device 20 for providing a visual representation of the field ofview of camera 16 a. Processor 36, as is further described below, canfurther analyze the captured image data for providing alerts (e.g.,audible and/or visual alerts, which can be partially or entirelygenerated in conjunction with alerting module 39) that are generated bydisplay device 20 and/or other components of the aircraft, such asaircraft avionics 34. In certain examples, processor 36 can receiveinputs from aircraft avionics 34 corresponding to, e.g., a phase offlight of the aircraft and/or a state of one or more aircraftcomponents, such as a weight-on-wheels input, aircraft airspeed,aircraft altitude, engine operating parameters, or other aircraft statevariables. Processor 36 can, in some examples, utilize the receivedinputs during processing of the image data captured by camera 16 a, asis further described below.

Camera 16 a can therefore provide image data captured from a field ofview of camera 16 a that is provided to display device 20 for a visualrepresentation of the field of view. The captured image data can befurther analyzed by processor 36 to provide alerts (such as, forexample, in conjunction with alerting module 39), guidance, or otheroutput to display device 20, aircraft avionics, or other aircraftcomponents to reduce crew workload, thereby increasing efficiency of theflight crew and enhancing operational safety of the aircraft.

FIG. 6 is an overhead view of aircraft 10 with cameras 16 a, 16 b, 16 cand 16 d having fields of view 38 a, 38 b, 38 c, and 38 d correspondingto cameras 16 a, 16 b, 16 c and 16 d, respectively. Cameras 16 a, 16 b,16 c, and 16 d can each be disposed within a different aircraft door ofaircraft 10. While the example of FIG. 6 is described herein withrespect to four cameras (i.e., cameras 16 a, 16 b, 16 c, and 16 d), inother examples, more than four cameras can be utilized or fewer thanfour cameras can be utilized.

As illustrated in FIG. 6 , cameras 16 a, 16 b, 16 c, and 16 d can beoriented such that each of cameras 16 a, 16 b, 16 c, and 16 d includes aunique field of view. The fields of view, as illustrated in FIG. 6 , caninclude overlapping portions. Cameras 16 a, 16 b, 16 c, and 16 d can bedisposed and oriented at various doors and locations about aircraft 10(including both sides of aircraft 10) such that a combined field of viewamong the set of cameras covers a substantial portion of thecircumference of aircraft 10, such as eighty percent, ninety percent, oreven an entirety of a circumference of aircraft 10. As described herein,captured image data corresponding to the fields of view can beaggregated to produce a combined image that represents an aggregate ofthe fields of view and such that image data from overlapping fields ofview is presented only once in the combined image.

As illustrated in FIG. 6 , field of view 38 a (corresponding to camera16 a) overlaps a portion of field of view 38 b (corresponding to camera16 b) at a minimum, and may overlap field of view 38 c (corresponding tocamera 16 c), but may not overlap field of view 38 d (corresponding tocamera 16 d). Field of view 38 b overlaps a portion of field of view 38a, field of view 38 c, and may overlap field of view 38 d, though neednot in all examples. Field of view 38 c may overlap a portion of fieldof view 38 a and at least a portion of field of view 38 b and field ofview 38 d. Field of view 38 d may overlap a portion of field of view 38b and a portion of field of view 38 c, but does not necessarily overlapfield of view 38 a.

Cameras 16 a, 16 b, 16 c and 16 d can each be operatively connected toone or more processors (not illustrated in FIG. 6 ). In one example, theprocessor is a central processor that aggregates the image data andoutputs the combined image data for display at one or more displaydevices. For instance, each of cameras 16 a, 16 b, 16 c, and 16 d can beoperatively connected to a separate display device via one or moreprocessors that receives the aggregated image data and outputs thecombined image for display at the respective display device. In otherexamples, the aggregated image data can be output for display at asingle display device, such as a display device in the cockpit ofaircraft 10.

Accordingly, cameras 16 a, 16 b, 16 c, and 16 d can capture image datafrom overlapping fields of view. The captured image data can beaggregated and presented in a combined image that represents anaggregate of the fields of view of the set of cameras 16 a, 16 b, 16 c,and 16 d. The combined image can be displayed within aircraft 10,thereby providing a single image of the external environment of aircraft10 that enhances situational awareness of the flight crew.

Wing Monitoring for Anticipated Foreign Object Collisions

FIG. 7A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning of anticipated collision based on an identified regionwithin the captured image data that corresponds to an edge of a wing ofthe aircraft. FIG. 7B illustrates aircraft 10 with aircraft skin 12,including doors 14 a, 14 b, and 14 c having cameras 16 a (with field ofview F_(a)), 16 b (with field of view F_(b)), and 16 c (with field ofview F_(c)) disposed therein, object O, damaged area D, and iceaccretion I. FIGS. 7A-7B will be discussed together. For purposes ofclarity and ease of discussion, the example operations of FIG. 7A aredescribed below within the context of camera 16 b (FIGS. 1-6 and 7B)disposed within aircraft door 14 b (FIGS. 1-4 and 7B) and operativelyconnected to display device 20 (FIGS. 2 and 5 ).

As described in more detail below, processor 36 can analyze the imagedata captured by camera 16 b in a variety of ways by monitoring theleading edge of a wing of aircraft 10. This monitoring can include, forexample, monitoring field of view F_(b) of camera 16 b for foreignobjects approaching the leading edge of the wing and/or detectingdeformation (due to damage or ice accretion) on the leading edge of thewing. Processor 36 can further be configured to produce a warningassociated with the leading edge of the wing in response to the capturedimage data from camera 16 b. This warning can communicate that, forexample, a collision with a foreign object is likely and/or that thewing is deformed as compared to a baseline state (as described below inreference to FIGS. 7B-8 ).

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 40). For example, processor 36 can receiveimage data captured by camera 16 b having field of view F_(b) that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 42). For instance, processor 36 canoutput the captured image data that is received from camera 16 b fordisplay at display device 20 that is disposed within an interior ofaircraft 10.

A region within the captured image data that corresponds to an edge of awing of the aircraft is identified (Step 44). For example, camera 16 bcan be disposed within a door of aircraft 10 such that field of viewF_(b) of camera 16 b is oriented to capture image data corresponding toa wing of aircraft 10. Processor 36 can analyze the captured image datato identify a region of the captured image data that corresponds to anedge of the wing of aircraft 10. For instance, processor 36 can utilizean edge detection algorithm to identify edges of the wing of aircraft10.

As one example, processor 36 can utilize the Canny edge detectormulti-stage algorithm to identify and track strong edges within theimage of the captured image data associated with the wing of theaircraft, though other edge detection algorithms are possible. The Cannyedge detector algorithm smooths the captured image data (e.g., via aGaussian filter) to remove noise, determines intensity gradients of thecaptured image data (e.g., via intensity values associated with eachpixel of the image), and removes spurious edge detection using a lowerbound cut-off (or other magnitude threshold) of gradient magnitudes.Thereafter, the Canny edge detector algorithm categorizes pixelsassociated with potential edges as one of a strong edge, a weak edge, ora suppressed pixel based on a comparison of the magnitude of thegradient associated with the potential edge pixel to threshold values.Those pixels associated with a gradient that is greater than an upperthreshold value are categorized as strong edge pixels. Those pixelsassociated with a gradient that is less than the upper threshold valuebut greater than a lower threshold value are categorized as weak edgepixels. Those pixels associated with a gradient that is less than thelower threshold value are categorized as suppressed pixels. Strong edgepixels are included in the candidate edge. Weak edge pixels are includedin the candidate edge if a strong edge pixel is included within aneight-pixel neighborhood of the weak edge pixel. Suppressed pixels aswell as weak edge pixels that are not within an eight-pixel neighborhoodof a strong edge pixel are not included in the candidate edge.

In some examples, processor 36 categorizes a region of pixels about theidentified edge as corresponding to the edge of the wing. For instance,a leading edge of the wing, depending on the vantage point and the fieldof view of the camera, can be categorized based on a region of pixelsrather than a line of pixels to thereby accommodate the rounded edge ofthe leading edge of the wing. Processor 36 identifies edges of the wingand regions corresponding to the wing for use in determining relativemotion of the wing through successive image frames in the captured imagedata and for identifying potential collisions.

A motion vector of the identified region within the captured image datathat corresponds to the edge of the wing of the aircraft is determined(Step 46). For example, processor 36 can determine a motion vectorassociated with the region within the captured image data received fromcamera 16 a that corresponds to the wing of the aircraft using multipleframes of image data received from camera 16 a. For instance, processor36 can utilize an Optical Flow algorithm, such as the Horn-Shunckmethod, the Lucas Kanade method, the Pyramid-KL algorithm, or otheroptical flow algorithm to generate a motion (or velocity) vectorcorresponding to pixels in the region of the captured image datacorresponding to the identified edge of the wing of the aircraft. SuchOptical Flow algorithms utilize a change of an identified pixel in theframe sequence of image data and correlation between adjacent frames tocorrelate pixels between frames and to determine motion information,including a motion (or velocity) vector between frames. Processor 36 candetermine a motion vector associated with the identified regioncorresponding to the edge of the wing of the aircraft as an average (orother central tendency) of the direction and magnitude of motion vectorsdetermined for the pixels included in the identified edge of the wing.

A region within the captured image data that corresponds to an object,such as object O in FIG. 7B, that is separate from the aircraft isidentified (Step 48). For example, processor 36 can analyze the capturedimage data by utilizing an object detection algorithm, such as theSingle Shot Detector (SSD) algorithm for object detection, the You OnlyLook Once (YOLO) object recognition algorithm, or other real-time objectdetection algorithm to identify a region of the captured image datacorresponding to an object separate from aircraft 10, such as object O.

As an example, processor 36 can utilize the YOLO object recognitionalgorithm to identify a region within the captured image datacorresponding to object O as the region within the captured image datathat is output by the YOLO algorithm as a bounding box around anidentified object. For instance, the YOLO algorithm (a real-time neuralnetwork-based algorithm) can be trained using baseline image data ofobjects to recognize any one or more of a plurality of objects.Candidate objects can include, e.g., humans, vehicles of various typeand size, jetways, buildings, aircraft, walls, or other objects that maybe encountered by an aircraft during, e.g., taxiing, docking, or otheroperations. The YOLO algorithm divides an image into regions andproduces bounding boxes in relation to the image data that encloseidentified objects. Processor 36, executing a YOLO algorithm forexample, can determine a region of the captured image data correspondingto an object that is separate from the aircraft as a region of abounding box surrounding an object that is produced by the YOLOalgorithm. In some examples, a library of common airborne or surfaceobjects (such as birds, poles, buildings, fences, aircraft wings ortails, and ground vehicles) can be maintained to enable real-timeidentification of objects as compared against the library database. Thisimage library can assist in proactive identification of potentialcollision objects and present pre-emptive warnings to crew members.

A motion vector of the region that corresponds to the object separatefrom the aircraft, such as object O, within the captured image data isdetermined (Step 50). For instance, processor 36 can utilize an OpticalFlow algorithm as was described above to generate a motion (or velocity)vector corresponding to pixels in the region of the captured image datacorresponding to the bounding box surrounding the identified object.Processor 36 can determine the motion vector of the object separate fromthe aircraft as an average or other central tendency of the directionand magnitudes of the motion vectors associated with the pixels of thebounding box. In the example shown in FIG. 7B, paths P₁ and P₂ are shownas motion vectors for object O.

An anticipated future collision location within the captured image datais determined based on the motion vector of the region that correspondsto the edge of the wing and the motion vector that corresponds to theobject (Step 52). For example, processor 36 can utilize the motionvector corresponding to the edge of the wing and the motion vectorcorresponding to the object to determine whether the object and the edgeof the wing are anticipated to intersect within the captured image data.If object O is traveling along path P₁, a future collision with the wingis anticipated, and a warning of anticipated future collision will beproduced in Step 54. If object O is traveling along path P₂, a futurecollision is unlikely, and a warning of anticipated collision will notbe produced.

A warning of anticipated collision based on the anticipated futurecollision location is produced (Step 54). For instance, processor 36 canoutput a visual alert for display at display device 20 and/or a separatedisplay device within aircraft 10 (e.g., an EFIS display). In certainexamples, such as when display device 20 includes a speaker device,processor 36 can cause display 20 (or other audio output device) toproduce an audible alarm corresponding to the anticipated futurecollision. In some examples, processor 36 can output an alertnotification (e.g., a status or other notification) to aircraft avionicsor other aircraft systems via an aircraft communication data bus. Insome examples, the warning of anticipated collision can includeinstructions to crew members regarding how to avoid the object separatefrom the aircraft. Camera 16 b and processor 36 can form part of asystem which can record captured image data for playback. This can allowfootage of objects separate from the aircraft to be cached temporarilyand/or stored long-term. This system can display footage of objectsand/or collisions for crew members and can be used as a library ofcommonly encountered foreign objects.

Accordingly, processor 36 that is operatively connected with camera 16 acan analyze captured image data received from camera 16 b to identify ananticipated collision between a wing of the aircraft and an objectseparate from the aircraft. Processor 36 can output a warning of theanticipated collision, thereby alerting the pilots or other flight crewand increasing safety of operation of aircraft 10. Captured image datafrom any of cameras 16 a, 16 b, 16 c, 16 d can be combined to gatheradditional information about the external environment of aircraft 10.Additionally, a system as described above provides numerous advantages.Camera 16 b and processor 36 can enable the detection of objects priorto collision, view or detect the collision as it occurs, and assesscharacteristics of any damage to determine a next potential course ofaction for crew members.

Wing Monitoring for Deformation Compared to Baseline Data

FIG. 8 is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning in response to determining that a region within thecaptured image data that corresponds to a leading edge of a wing of theaircraft does not conform to baseline image data. For purposes ofclarity and ease of discussion, the example operations of FIG. 8 aredescribed below within the context of camera 16 b (FIGS. 1-6 and 7B)disposed within aircraft door 14 a (FIGS. 1-4 and 7B) and operativelyconnected to display device 20 (FIGS. 2 and 5 ).

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 56). For example, processor 36 can receiveimage data captured by camera 16 b having a field of view that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 58). For instance, processor 36 canoutput the captured image data that is received from camera 16 b fordisplay at display device 20 that is disposed within an interior ofaircraft 10.

A region within the captured image data that corresponds to a leadingedge of a wing of the aircraft is identified (Step 60). For example,camera 16 b can be disposed within a door of aircraft 10 such that thefield of view of camera 16 b is oriented to capture image datacorresponding to a leading edge of a wing of aircraft 10. Processor 36can analyze the captured image data to identify a region of the capturedimage data that corresponds to a leading edge of the wing of aircraft10. For instance, as was previously described above, processor 36 canutilize an edge detection algorithm, such as the Canny edge detectoralgorithm or other edge detection algorithm to identify a region withinimage data captured by camera 16 b that corresponds to a leading edge ofa wing of aircraft 10.

It is determined whether the region within the captured image data thatcorresponds to the leading edge of the wing conforms to baseline imagedata corresponding to the leading edge of the wing (Step 62).Non-conformance of captured image data with baseline image data canoccur when, for example, there is ice accretion (such as ice accretion Ishown in FIG. 7B) or foreign object damage (such as damaged area D shownin FIG. 7B) on at least a portion of the wing. For example, processor 36can access pixel coordinates of baseline image data associated with aleading edge of a wing of aircraft 10 captured by camera 16 b in abaseline (e.g., known healthy) state. For instance, camera 16 b cancapture baseline image data of the wing of aircraft 10 when aircraft 10is in a known, healthy state (e.g., without ice accretion on the wing,without deformation corresponding to damage of the wing, or otherwise ina known, healthy state). This baseline image data can be captured when,for example, aircraft door 14 b is opened and closed during boardingprocedures. This allows camera 16 b to capture multiple views of thewing and calibrate the baseline image data. Processor 36 can analyze thebaseline image data to identify a leading edge of the wing, such as viathe Canny edge detector algorithm or other edge detection algorithm.Processor 36 can cause computer-readable memory 37 to store pixelcoordinates corresponding to the leading edge of the wing that areidentified based on the baseline image data.

Processor 36 can compare the pixel coordinates associated with theleading edge of the wing that are identified from the captured imagedata received from camera 16 b during operation with the stored pixelcoordinates corresponding to the baseline image data. Processor 36 candetermine, in some examples, that the region that corresponds to theleading edge of the wing within the captured image data received fromcamera 16 b does not conform to the baseline image data in response todetermining that the pixel coordinates associated with the leading edgeof the wing in the captured image data received from camera 16 bdeviates from the stored pixel coordinates associated with the baselinedata by a threshold deviation. For instance, processor 36 can generate afirst vector of pixel coordinates associated with the leading edge ofthe wing that are identified from the captured image data received fromcamera 16 b. Processor 36 can generate a second vector of pixelcoordinates as the stored pixel coordinates corresponding to thebaseline image data. Processor 36 can determine an angle between thefirst vector and the second vector, the angle representing an extent ofdeviation between the two edges. That is, an angle of zero between thetwo vectors represents an identical match of pixel coordinates betweenthe two edges. An increased angle between the first vector and thesecond vector corresponds to an increased extent of deviation betweenthe first vector and the second vector.

Processor 36 can determine that the region within the captured imagedata that corresponds to the leading edge of the wing does not conformto the baseline image data corresponding to the leading edge of the wingin response to determining that the angle between the first vector andthe second vector exceeds a threshold angle. Such deviation canrepresent ice accretion (such as ice accretion I shown in FIG. 7B) onthe leading edge of the wing, deformation of the leading edge of thewing (e.g., due to foreign object damage or other damage; such asdamaged area D shown in FIG. 7B), or other physical change to theleading edge of the wing that could impact aerodynamic performance ofthe wing. Camera 16 b and processor 36, which have been calibrated onthe wing surface contour, form, and dimensionality as described above,can assess that damage/deformation to the wing has occurred, determine alocation of the damage/deformation, and determine an approximate size ofthe damaged/deformed area. Camera 16 b, or cameras 16 a, 16 c, 16 d, canadditionally or alternatively monitor the trailing edge of the wing todetect damage or deformation. This monitoring can additionally oralternatively determine damage to, or improper movement of, wing controlsurfaces.

A warning associated with the leading edge of the wing is produced andoutput in response to determining that the region within the image datathat corresponds to the leading edge of the wing does not conform to thebaseline image data (Step 64). For example, processor 36 can output avisual alert for display at display device 20 and/or a separate displaydevice within aircraft 10 (e.g., an EFIS display). In certain examples,such as when display device 20 includes a speaker device, processor 36can cause display 20 (or other audio output device) to produce anaudible alarm corresponding to the warning associated with the leadingedge of the wing. In some examples, processor 36 can output an alertnotification (e.g., a status or other notification) to aircraft avionicsor other aircraft systems via an aircraft communication data bus. Thewarning associated with the leading edge of the wing can include anestimation of the location and size of the damaged area.

Accordingly, techniques of this disclosure can enable processor 36 toanalyze captured image data received from camera 16 b to produce awarning in response to determining that a leading edge of the wing doesnot conform to a baseline (e.g., known, healthy) state, thereby alertingthe pilots or other flight crew and increasing safety of operation ofaircraft 10. Additionally, a system as described above provides numerousadvantages. Camera 16 b and processor 36 can enable the detection ofobjects prior to collision, view or detect the collision as it occurs,and assess characteristics of any damage to determine a next potentialcourse of action for crew members.

Engine Inlet Monitoring for Ground Personnel Safety Clearance

FIG. 9A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning corresponding to proximity of ground personnel to anengine inlet. FIG. 9B illustrates aircraft 10 with aircraft skin 12,including doors 14 a, 14 b, and 14 c having cameras 16 a (with field ofview F_(a)), 16 b (with field of view F_(b)), and 16 c (with field ofview F_(c)) disposed therein, group personnel GP₁ and GP₂, and thresholdregion T. FIGS. 9A-9B will be discussed together. For purposes ofclarity and ease of discussion, the example operations of FIG. 9A aredescribed below within the context of camera 16 b (FIGS. 1-6 and 9B)disposed within aircraft door 14 b (FIGS. 1-4 and 9B) and operativelyconnected to display device 20 (FIGS. 2 and 5 ).

As described in more detail below, processor 36 can analyze the imagedata captured by camera 16 b in a variety of ways by monitoring theengine inlet of an engine of aircraft 10. This monitoring can include,for example, monitoring field of view F_(b) of camera 16 b for foreignobjects approaching the engine inlet and/or detecting deformation (dueto damage or ice accretion) on the engine inlet. Processor 36 canfurther be configured to produce a warning associated with the engineinlet in response to the captured image data from camera 16 b. Thiswarning can communicate that, for example, ground personnel are within athreshold distance of the engine inlet (as described in reference toFIGS. 9A-9B), ingestion of a foreign object is likely (as describedbelow in reference to FIGS. 10A-10B), and/or that the engine inlet isdeformed as compared to a baseline state (as described below inreference to FIGS. 11A-11B).

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 66). For example, processor 36 can receiveimage data captured by camera 16 b having field of view F_(b) that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 68). For instance, processor 36 canoutput the captured image data that is received from camera 16 b fordisplay at display device 20 that is disposed within an interior ofaircraft 10.

A region within the captured image data that corresponds to an engineinlet of an engine on the aircraft is identified (Step 70). For example,processor 36 can analyze the captured image data by utilizing an objectdetection algorithm, such as the SSD algorithm for object detection, theYOLO object recognition algorithm, or other real-time object detectionalgorithm to identify a region of the captured image data correspondingto an engine inlet. For instance, processor 36 can utilize the YOLOalgorithm that has been trained via image data of the inlet of theengine of aircraft 10 to recognize the engine inlet as an object. Inother examples, a different object detection algorithm can be used.Processor 36, executing, e.g., the YOLO algorithm, can identify theregion within the captured image data that corresponds to the engineinlet on aircraft 10 as the region of the bounding box produced by theYOLO algorithm that encloses the identified engine inlet object withinthe captured image data.

A region within the captured image data that corresponds to a personoutside the aircraft, such as ground personnel GP₁, GP₂, is identified(Step 72). For instance, processor 36 can execute the Histogram ofOriented Gradients (HOG) algorithm, the YOLO object recognitionalgorithm, the SSD algorithm, or other object detection algorithmtrained on image data of humans to identify a person as an object withinthe captured image data. For instance, processor 36 can execute the YOLOobject recognition algorithm and can identify the bounding box enclosingan identified object corresponding to a person within the captured imagedata as the region within the captured image data that corresponds tothe person outside the aircraft.

An image distance between the region within the captured image data thatcorresponds to the engine inlet and the region within the captured imagedata that corresponds to the person outside the aircraft is determined(Step 74). For instance, processor 36 can determine, e.g., ashortest-path least number of pixels between the region of the capturedimage data that corresponds to the engine inlet and the region of thecaptured image data that corresponds to the person outside the aircraft.The image distance is converted to a physical distance based on thefield of view of the camera (Step 76). For example, processor 36 canconvert the number of pixels corresponding to the image distance betweenthe regions based on a known distance between the mounting location ofcamera 16 b and the engine inlet and a known number of pixelscorresponding to the distance.

A warning corresponding to proximity of ground personnel to the engineinlet is produced in response to determining that the physical distanceis less than a threshold distance (Step 78). In FIG. 9B, the thresholddistance is defined by the threshold region T, and ground personnel GP₁is within the region T (and accordingly, less than the thresholddistance from the engine inlet) while ground personnel GP₂ is outsidethe threshold region T (more than the threshold distance from the engineinlet). In the example shown in FIG. 9B, ground personnel GP₁ wouldtrigger a warning, while ground personnel GP₂ would not trigger awarning. The defined threshold distance can be based on a region (suchas region T) defined by selected or calculated boundary locations.Additionally or alternatively, the threshold distance can itself beselected or calculated. Multiple threshold distances and/or regions canbe selected for different gradations of possible alerts. For example, asecondary threshold distance and/or region which is larger than aprimary threshold distance and/or region can be defined which triggersan alert to warn ground personnel that they are nearing the primarythreshold distance and/or region. The threshold distance and/or thethreshold region can vary based on parameters such as engine speedand/or power level. For example, a larger threshold distance and/orregion can be used for a high engine power level than for a low enginepower level. Processor 36 can output a visual alert for display atdisplay device 20 and/or a separate display device within aircraft 10(e.g., an EFIS display). In certain examples, such as when displaydevice 20 includes a speaker device, processor 36 can cause display 20(or other audio output device) to produce an audible alarm correspondingto the warning associated with the proximity of ground personnel to theengine inlet. In some examples, processor 36 can output an alertnotification (e.g., a status or other notification) to aircraft avionicsor other aircraft systems via an aircraft communication data bus.Processor 36 can additionally or alternatively wirelessly alert devicesoutside of aircraft 10, such as devices worn or held by groundpersonnel.

Accordingly, techniques of this disclosure can enable processor 36 toanalyze captured image data received from camera 16 b to produce awarning in response to determining that a ground personnel is within athreshold distance from an engine inlet of an engine of aircraft 10,thereby alerting the pilots or other flight crew and increasing safetyof operation of aircraft 10. Captured image data from any of cameras 16a, 16 b, 16 c, 16 d can be combined to gather additional informationabout the external environment of aircraft 10.

Engine Inlet Monitoring for Anticipated Foreign Object Ingestion

FIG. 10A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning corresponding to ingestion of a foreign object to anengine inlet of an engine of the aircraft. FIG. 10B illustrates aircraft10 with aircraft skin 12, including doors 14 a, 14 b, and 14 c havingcameras 16 a (with field of view F_(a)), 16 b (with field of viewF_(b)), and 16 c (with field of view F_(c)) disposed therein, and objectO. FIGS. 10A-10B will be discussed together. For purposes of clarity andease of discussion, the example operations of FIG. 10A are describedbelow within the context of camera 16 b (FIGS. 1-6 and 10B) disposedwithin aircraft door 14 b (FIGS. 1-4 and 10B) and operatively connectedto display device 20 (FIGS. 2 and 5 ).

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 80). For example, processor 36 can receiveimage data captured by camera 16 b having field of view F_(b) that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 82). For instance, processor 36 canoutput the captured image data that is received from camera 16 b fordisplay at display device 20 that is disposed within an interior ofaircraft 10. In some examples, the display device can have the abilityto continuously watch for a potential object ingestion, alert crewmembers of a potential object ingestion, and/or show crew members videoof an object ingestion.

A region within the captured image data that corresponds to an engineinlet of an engine on the aircraft is identified (Step 84). For example,processor 36 can analyze the captured image data by utilizing an objectdetection algorithm, such as the SSD algorithm for object detection, theYOLO object recognition algorithm, or other real-time object detectionalgorithm to identify a region of the captured image data correspondingto an engine inlet. For instance, processor 36 can utilize the YOLOalgorithm that has been trained via image data of the inlet of theengine of aircraft 10 to recognize the engine inlet as an object, thoughin other examples, a different object detection algorithm can be used.Processor 36, executing, e.g., the YOLO algorithm, can identify theregion within the captured image data that corresponds to the engineinlet on aircraft 10 as the region of the bounding box produced by theYOLO algorithm that encloses the identified engine inlet object withinthe captured image data.

A region within the captured image data that corresponds to an objectseparate from the aircraft, such as object O in FIG. 10B, is identified(Step 86). For example, processor 36 can analyze the captured image databy utilizing an object detection algorithm, such as the SSD algorithmfor object detection, the YOLO object recognition algorithm, or otherreal-time object detection algorithm to identify a region of thecaptured image data corresponding to an object separate from aircraft10. For instance, processor 36 can utilize the YOLO object recognitionalgorithm to identify the region within the captured image datacorresponding to an object as the region within the captured image datathat is output by the YOLO algorithm as a bounding box around anidentified object, though other object detection algorithms arepossible. For instance, the YOLO algorithm can be trained using baselineimage data of objects to recognize any one or more of a plurality ofobjects. Candidate objects can include, e.g., birds of various type andsize, and/or shapes of various type and size, such as ovals, circles,squares, or other polygons that may be encountered by an aircraft inflight or during other phases of operation. Processor 36, executing aYOLO algorithm for example, can determine a region of the captured imagedata corresponding to an object that is separate from the aircraft as aregion of a bounding box surrounding an object that is produced by theYOLO algorithm.

A trajectory of the region that corresponds to object O relative to theregion within the captured image data that corresponds to the engineinlet is determined (Step 88). For instance, processor 36 can determinethe trajectory of the identified object based on a relative locationwithin the captured image data of the object within successive frames ofthe captured image data. In other examples, processor 36 can determinethe trajectory of the identified object based on a location of theidentified object within the captured image data and a motion vector ofthe identified object determined based on, e.g., an Optical Flow orother motion tracking algorithm.

It is determined, based on the trajectory of the identified object, thata probability that the object was ingested by the engine inlet exceeds athreshold probability (Step 90). For example, processor 36 candetermine, based on the trajectory of the identified object and alocation of region of the captured image data corresponding to theengine inlet, a probability that the identified object intersected theengine inlet and was ingested by the engine inlet. Processor 36 cancompare the determined probability to a threshold probability, such as afifty percent probability, a sixty percent probability, or otherthreshold probability. In the example shown in FIG. 10B, paths P₁ and P₂are shown as trajectories for object O. If object O is traveling alongpath P₁, a future ingestion is likely, and a warning corresponding toingestion will be produced in Step 92. If object O is traveling alongpath P₂, a future ingestion is unlikely, and a warning corresponding toingestion will not be produced.

A warning corresponding to ingestion of a foreign object to the engineinlet is produced in response to determining that probability that theobject was ingested by the engine inlet exceeds the thresholdprobability (Step 92). For example, processor 36 can output a visualalert for display at display device 20 and/or a separate display devicewithin aircraft 10 (e.g., an EFIS display). In certain examples, such aswhen display device 20 includes a speaker device, processor 36 can causedisplay 20 (or other audio output device) to produce an audible alarmcorresponding to the warning associated with the ingestion of a foreignobject to the engine inlet. In some examples, processor 36 can output analert notification (e.g., a status or other notification) to aircraftavionics or other aircraft systems via an aircraft communication databus.

Accordingly, techniques of this disclosure can enable processor 36 toanalyze captured image data received from camera 16 a to produce awarning in response to determining that a probability that a foreignobject was ingested by the engine inlet exceeds a threshold probability,thereby alerting the pilots or other flight crew and increasing safetyof operation of aircraft 10. Additionally, a system as described aboveprovides numerous advantages. Camera 16 b and processor 36 can enablethe detection of objects prior to collision, view or detect thecollision as it occurs, and assess characteristics of any damage todetermine a next potential course of action for crew members.

Engine Inlet Monitoring for Deformation Compared to Baseline Data

FIG. 11A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning associated with an engine inlet in response todetermining that an identified region within the capture image data thatcorresponds to the engine inlet does not conform to baseline image data.FIG. 11B illustrates aircraft 10 with aircraft skin 12, including doors14 a, 14 b, and 14 c having cameras 16 a (with field of view F_(a)), 16b (with field of view F_(b)), and 16 c (with field of view F_(c))disposed therein, damaged area D, and ice accretion I. FIGS. 11A-11Bwill be discussed together. For purposes of clarity and ease ofdiscussion, the example operations of FIG. 11A are described belowwithin the context of camera 16 b (FIGS. 1-6 and 11B) disposed withinaircraft door 14 b (FIGS. 1-4 and 11B) and operatively connected todisplay device 20 (FIGS. 2 and 5 ).

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 94). For example, processor 36 can receiveimage data captured by camera 16 b having a field of view that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 96). For instance, processor 36 canoutput the captured image data that is received from camera 16 b fordisplay at display device 20 that is disposed within an interior ofaircraft 10.

A region within the captured image data that corresponds to an engineinlet of an engine on the aircraft is identified (Step 98). For example,processor 36 can analyze the captured image data by utilizing an objectdetection algorithm, such as the SSD algorithm for object detection, theYOLO object recognition algorithm, or other real-time object detectionalgorithm to identify a region of the captured image data correspondingto an engine inlet. For instance, processor 36 can utilize the YOLOalgorithm that has been trained via image data of the inlet of theengine of aircraft 10 to recognize the engine inlet as an object, thoughin other examples, a different object detection algorithm can be used.Processor 36, executing, e.g., the YOLO algorithm, can identify theregion within the captured image data that corresponds to the engineinlet on aircraft 10 as the region of the bounding box produced by theYOLO algorithm that encloses the identified engine inlet object withinthe captured image data.

It is determined whether the region within the image data thatcorresponds to the engine inlet conforms to baseline image datacorresponding to the engine inlet of the engine (Step 100).Non-conformance of captured image data with baseline image data canoccur when, for example, there is ice accretion (such as ice accretion Ishown in FIG. 11B) or foreign object damage (such as damaged area Dshown in FIG. 11B) on at least a portion of the engine inlet. Forexample, processor 36 can perform a strict comparison of color andintensity of pixels within the identified region of the captured imagedata that corresponds to the engine inlet and color an intensity ofpixels within the baseline image data of the engine inlet, such asbaseline image data utilized for training the YOLO algorithm torecognize the engine inlet as an object. In such an example, processor36 can determine that the image data within the region of the capturedimage data corresponding to the engine inlet does not conform to thebaseline image data in response to determining that a threshold numberof pixels deviate from the baseline image data.

In some examples, processor 36 can perform a correlation comparison ofthe region within captured image data that corresponds to the engineinlet and baseline image data of the engine inlet to provide anindication of an extent by which the region within the captured imagedata that corresponds to the engine inlet deviates from the baselineimage data. In such examples, processor 36 can determine that the imagedata within the region of the captured image data corresponding to theengine inlet does not conform to the baseline image data in response todetermining that the indication of the extent of the deviation exceeds athreshold value.

In yet other examples, processor 36 can utilize fuzzy pixel comparison,histogram comparison, and/or image masking techniques to determinewhether the image data within the region of the captured image datacorresponding to the engine inlet conforms to the baseline image data.

A warning associated with the engine inlet is produced in response todetermining that the region within the image data that corresponds tothe engine inlet does not conform to the baseline image data (Step 102).For example, processor 36 can output a visual alert for display atdisplay device 20 and/or a separate display device within aircraft 10(e.g., an EFIS display). In certain examples, such as when displaydevice 20 includes a speaker device, processor 36 can cause display 20(or other audio output device) to produce an audible alarm correspondingto the warning associated with the engine inlet. In some examples,processor 36 can output an alert notification (e.g., a status or othernotification) to aircraft avionics or other aircraft systems via anaircraft communication data bus.

Accordingly, techniques of this disclosure can enable processor 36 toanalyze captured image data received from camera 16 b to produce awarning in response to determining that an engine inlet does not conformto baseline image data corresponding to the engine inlet (e.g., in aknown, healthy state), thereby alerting the pilots or other flight crewand increasing safety of operation of aircraft 10. Additionally, asystem as described above provides numerous advantages. Camera 16 b andprocessor 36 can enable the detection of objects prior to collision,view or detect the collision as it occurs, and assess characteristics ofany damage to determine a next potential course of action for crewmembers.

Landing Gear Monitoring for Chock Blocks and Landing Gear Condition

FIG. 12A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce an output that indicates whether an identified region within thecaptured image data that corresponds to a wheel of main landing gear ofthe aircraft includes image data corresponding to a chock block. FIG.12B illustrates aircraft 10 with aircraft skin 12, including doors 14 a,14 b, and 14 c having cameras 16 a (with field of view F_(a)), 16 b(with field of view F_(b)), and 16 c (with field of view F_(c)) disposedtherein, and chock blocks 17. FIG. 12C illustrates the subject matter ofFIG. 12B and further illustrates nose landing gear 19 and main landinggear 21. Each of nose landing gear 19 and main landing gear 21 include awheel and a tire. FIGS. 12A-12C will be discussed together. For purposesof clarity and ease of discussion, the example operations of FIG. 12Aare described below within the context of camera 16 c (FIGS. 1-6 and12B-12C) disposed within aircraft door 14 c (FIGS. 1-4 and 12B-12C) andoperatively connected to display device 20 (FIGS. 2 and 5 ).

As described above in reference to FIG. 3 , any of cameras 16 a, 16 b,16 c, 16 d can be oriented to provide a field of view which includes thecomponent to be monitored (here, the landing gear). As described in moredetail below, processor 36 can analyze the image data captured by camera16 c in a variety of ways by monitoring landing gear of aircraft 10.This monitoring can include, for example, monitoring field of view F_(c)of camera 16 c for objects such as chock blocks and/or detecting whetherthe landing gear is fully extended, as well as the condition of thetires (inflated or flat). In the example shown in FIGS. 12B-12C, thetire of the main landing gear is fully inflated. Processor 36 canfurther be configured to produce a warning associated with the landinggear in response to the captured image data from camera 16 c. Thiswarning can communicate that, for example, a chock block is present nearthe landing gear and/or that the landing gear is not fully extended orthat the tire condition is suspect/not fully inflated (as describedbelow in reference to FIGS. 12A-12C).

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 104). For example, processor 36 can receiveimage data captured by camera 16 c having field of view F_(c) that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 106). For instance, processor 36 canoutput the captured image data that is received from camera 16 c fordisplay at display device 20 that is disposed within an interior ofaircraft 10.

A region within the captured image data that corresponds to a wheel of amain landing gear of the aircraft is identified (Step 108). For example,processor 36 can analyze the captured image data by utilizing an objectdetection algorithm, such as the SSD algorithm for object detection, theYOLO object recognition algorithm, or other real-time object detectionalgorithm to identify a region of the captured image data correspondingto a main landing gear of aircraft 10. For instance, processor 36 canutilize the YOLO algorithm that has been trained via image data of themain landing gear of aircraft 10 to recognize the main landing gear asan object, though in other examples, a different object detectionalgorithm can be used. Processor 36, executing, e.g., the YOLOalgorithm, can identify the region within the captured image data thatcorresponds to the main landing gear of aircraft 10 as the region of thebounding box produced by the YOLO algorithm that encloses the identifiedmain landing gear object within the captured image data.

It is identified whether the region within the captured image data thatcorresponds to the wheel of the main landing gear includes image datacorresponding to a chock block (Step 110). For example, processor 36 canutilize the YOLO, SSD, or other object detection algorithm trained onimage data of a chock block to identify a region of the captured imagedata corresponding to a chock block. A similar method can be used toidentify whether the landing gear, such as main landing gear 21, isfully extended, including whether the landing gear is extended at thecorrect angle. In response to determining that a region corresponding tothe chock block is identified in the captured image data, processor 36can determine whether the region corresponding to the chock block isproximate the region of the captured image data corresponding to themain landing gear, such as by comparing an image distance (e.g., anumber of pixels or other distance) between the region within thecaptured image data corresponding to the main landing gear and theregion within the captured image data corresponding to the chock blockis less than a threshold image distance. In response to determining thatthe image distance is less than the threshold image distance, processor36 can determine that the chock block is present at the main landinggear. In response to determining that the image distance is greater thanthe threshold image distance or that no object was detectedcorresponding to the chock block, processor 36 can determine that thechock block is not present at the main landing gear.

In some examples, processor 36 can utilize on object detectionalgorithm, such as the YOLO algorithm, the SSD algorithm, or otherobject detection algorithm that is trained on image data of the mainlanding gear with the chock block in place at the main landing gear(i.e., within the threshold distance to the main landing gear). In suchexamples, processor 36 can execute the object detection algorithm toidentify a region of the captured image data corresponding to the mainlanding gear of the aircraft with the chock block in place as an objectwithin the captured image data. In response to determining that theobject corresponding to the main landing gear with the chock block inplace is identified in the captured image data, processor 36 candetermine that the image data that corresponds to the wheel of the mainlanding gear includes the image data corresponding to the chock block.In response to determining that the object corresponding to the mainlanding gear with the chock block in place is not identified in thecaptured image data, processor 36 can determine that the image data thatcorresponds to the wheel of the main landing gear does not include theimage data corresponding to the chock block.

In examples where the wheel of the landing gear is monitored for thepresence of chock blocks, a chocked main landing gear output is producedthat indicates whether the region within the image data that correspondsto the wheel of the main landing gear includes the image datacorresponding to the chock block (Step 112). In examples where theextension of the landing gear is monitored, an output can be producedwhich communicates that the landing gear is not fully extended. Forexample, processor 36 can produce an output for display at displaydevice 20 (or other display device within aircraft 10) that indicateswhether the chock block is in place at the main landing gear. In someexamples, processor 36 can output the indication (e.g., a status orother notification) to aircraft avionics or other aircraft systems viaan aircraft communication data bus.

In certain examples, processor 36 can execute the operations of theexample of FIG. 12A in response to receiving an indication that aircraft10 is docked, such as an input from aircraft avionics 34 indicating thata phase of the aircraft operation indicates that aircraft 10 is docked.As such, processor 36, in some examples, can perform the operations todetermine whether the chock block is present at the main landing gearonly when the phase of aircraft operation indicates that a chock blockpresence is expected.

Accordingly, techniques of this disclosure can enable processor 36 toanalyze captured image data received from camera 16 c to produce anindication of whether a chock block is present at main landing gear ofthe aircraft, thereby alerting flight crew in those instances when thechock block presence is expected but is not detected. Additionallyand/or alternatively, an output can communicate to the flight crew thatthe landing gear is not fully extended and/or that a tire of the landinggear is not fully inflated. As such, the techniques of this disclosurecan increase safety of operation of aircraft 10. Additionally, doorcameras from the left and right sides of the aircraft, or multiple doorcameras along one side of the aircraft, could generated aggregated imagedata to, for example, compare the horizon, landing gear height, or otherparameters. Cameras such as camera 16 c can continuously monitor for anevent relating to the landing gear (such as presence or lack of chockblocks, incomplete landing gear extension, and/or a blown tire), andthis system can alert crew members and show footage of such an event.Camera 16 c and processor 36 can additionally or alternatively form partof a system which can monitor the landing gear and the edges and/or edgelines of the ground surface to, for example, alert crew members to thepossibility of a potential excursion from the runway or taxi surface.This can allow crew members to avoid an excursion off of the runway ortaxi surface onto, for example, an unpaved soil surface. Mostsimplistically, this system can continuously monitor the main landinggear and runway/taxiway edges/edge lines to determine convergence ordivergence and, through aspect ratio analysis or other techniques,determine a magnitude of said convergence to determine a level ofalerting that is needed. Looking out both sides of the aircraft ontofinite width runway/taxiway surfaces, the combination of convergence onone side of the aircraft in lock step with the divergence on theopposite side of the aircraft can provide concurrence of a given levelof probability of excursion. Likewise, multiple door cameras placedalong the fuselage on a given aircraft side can work in concert to gaugethe turn vector of the aircraft relative to the detected edge conditionsand alert on predicted intersections at some distance along the surface.

Jet Bridge Alignment Monitoring

FIG. 13A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door tooutput an indication of an alignment state between the aircraft door anda cabin of a jet bridge, such as cabin 210 of jet bridge 200 (both shownin FIGS. 13B-13D). FIG. 13B is a front perspective view of aircraft 10,including a schematic depiction of field of view F_(a) of camera 16 a(shown in FIG. 1 ), and jet bridge 200 which includes cabin 210. FIG.13C illustrates jet bridge 200, including boundaries B of cabin 210,when cabin 210 is in an alignment state with aircraft 10. FIG. 13Dillustrates jet bridge 200, including boundaries B of cabin 210, whencabin 210 is not in an alignment state with aircraft 10. FIG. 13E is aschematic depiction of display device 20 outputting a graphical overlayof target and extracted alignment features. FIGS. 13A-13E will bediscussed together. For purposes of clarity and ease of discussion, theexample operations of FIG. 13A are described below within the context ofcamera 16 a (FIGS. 1-6 ) disposed within aircraft door 14 a (FIGS. 1-4 )and operatively connected to display device 20 (FIGS. 2 and 5 ).

In certain examples, the operations of the example of FIG. 13A can beexecuted in response to receiving an indication that aircraft 10 is in adocking phase, such as an input from aircraft avionics 34 indicatingthat a phase of the aircraft operation indicates that aircraft 10 isdocking (that is, within a docking distance of cabin 210 of jet bridge200). As such, the example operations, in certain examples, can beperformed only when the phase of aircraft operation indicates that a jetbridge alignment is expected.

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 114). For example, processor 36 can receiveimage data captured by camera 16 a having a field of view that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 116). For instance, processor 36 canoutput the captured image data that is received from camera 16 a fordisplay at display device 20 that is disposed within an interior ofaircraft 10.

Physical characteristics of cabin 210 of jet bridge 200 within thecaptured image data (here, field of view F_(a)) are identified (Step118). Physical characteristics of cabin 210 can include, for example,boundaries B (shown in FIGS. 13C-13D) of cabin 210 or other physicalcharacteristics of cabin 210. Processor 36 can identify the physicalcharacteristics within the captured image data, such as boundaries B ofcabin 210, using an edge detection algorithm. For instance, processor 36can utilize the Canny edge detector or other edge detection algorithm toidentify edges corresponding to outer boundaries B of cabin 210 withinthe captured image data. Another example of a physical characteristic ofcabin 210 can be a light level present within cabin 210. The brightnessof light within cabin 210 is lower than the amount of ambient lightoutside of cabin 210 (sunlight or floodlights used to illuminate theexternal environment of aircraft 10). When aircraft door 14 a issufficiently aligned and mated with cabin 210, this bright ambient lightwill not be within field of view F_(a) of camera 16 a. Processor 36 candetermine whether a light level experienced by camera 16 a exceeds anexpected threshold defined by the light levels within cabin 210.

Alignment features corresponding to the physical characteristics ofcabin 210 that are indicative of alignment between cabin 210 and theaircraft door are extracted from the captured image data using theidentified physical characteristics of jet bridge 200 (Step 120).Alignment features can include, e.g., relative orientation of identifiededges (such as boundaries B) of cabin 210 (indicating, e.g., a skewedorientation between the door and the cabin of the jet bridge), size ofthe identified edges within the captured image data (indicating, e.g.,distance of the aircraft door to the jet bridge), relative size of theidentified edges among the group of identified edges within the capturedimage data (indicating, e.g., a skewed orientation), or other featuresthat are indicative of alignment between cabin 210 and the aircraftdoor.

It is determined, based on the alignment features, whether the physicalcharacteristics of cabin 210 within the captured image data satisfythreshold alignment criteria to produce an alignment state (Step 122).An example of an alignment state is illustrated in FIG. 13C, while FIG.13D illustrates an example where an alignment state is not present. Forexample, an alignment model (e.g., a linear regression or other model)can be developed and trained using machine learning or other techniquesto produce an indication of an extent by which the extracted alignmentfeatures correlate to alignment features extracted from baseline (ortraining) image data captured by camera 16 a while cabin 210 is alignedwith the aircraft door. Processor 36 can extract the alignment featuresfrom the captured image data received from camera 16 a and can utilizethe trained alignment model to produce an output that indicates anextent by which the extracted alignment features correlate with theextracted features from the baseline image data, such as a normalizedvalue (e.g., between a value of zero and one, between a value of zeroand one hundred, or other normalized value) that indicates the extent ofalignment. Processor 36 can compare the output that indicates the extentof alignment to a threshold alignment value to produce an alignmentstate that indicates whether the cabin of the jet bridge is aligned withthe door of aircraft 10.

An indication of the alignment state is output (Step 124). For instance,processor 36 can output an indication of the alignment state for displayat display device 20 or other display device. In certain examples, suchas the example shown in FIG. 13E, the processor 36 can output theindication of the alignment state as a graphical overlay displayed atdisplay device 20 of target alignment features over a graphicalindication of the identified alignment features extracted from thecaptured image data. In the example shown in FIG. 13E, aircraft 10 isnot in an alignment state with cabin 210, and accordingly a differenceis demonstrated on the graphical overlay between the target andextracted alignment features. Another example of an indication ofalignment state can be a green/red light system which indicates to crewmembers, such as a jet bridge operator, whether aircraft 10 ismisaligned with cabin 210 (for example, if aircraft 10 is likely to hitcabin 210). A green light indicates that jet bridge alignment isproceeding in such a way that no corrections are needed, while a redlight indicates that a correction is needed to avoid a misalignmentand/or collision with cabin 210. The green/red light indication canadditionally or alternatively be wirelessly sent to crew members who arenot inside aircraft 10. Another example of an indication of alignmentstate can be an alert which is triggered if the aircraft door is openand cabin 210 of jet bridge 200 moves away from aircraft 10 (that is,out of a mated position with the aircraft door), and/or which istriggered if cabin 210 is not aligned with aircraft door 14 a and anattempt is made to open aircraft door 14 a.

Accordingly, processor 36 can analyze captured image data received fromcamera 16 a to produce an indication of an alignment state between thecabin of the jet bridge and the aircraft door, thereby indicating thealignment state to flight crew or other personnel an increasingefficiency of docking operations of aircraft 10. Additionally, theperspective of a door camera such as camera 16 a provides numerousadvantages over a camera in another location on or off aircraft 10.Camera 16 a is able to provide a view of the inside of cabin 210 of jetbridge 200, while cameras in other locations would be able to provideonly a view of the outside of cabin 210. A view of the inside of cabin210 can provide more consistent alignment fiducials than an externalview of cabin 210. The position of camera 16 a can allow for themonitoring of other alignment features, such as whether field of viewF_(a) includes bright ambient light. The use of camera 16 a can improvethe visibility and image quality available to crew members overconventional CCTV. A green/red light system can increase the efficiencyand confidence of crew such as jet bridge operators and avoid damage tocabin 210 and/or aircraft 10. Camera 16 a can form part of a system withaircraft door 14 a to facilitate readiness for door opening and/orfacilitate automated aircraft door opening processes when the correctsignals are present (in a similar manner to the green/red light systemdescribed above, the system could receive an all-clear light whichallows the aircraft doors to open). Finally, this system can facilitatewarnings to crew members if aircraft door 14 a is opened or opening whencabin 210 is not aligned with aircraft door 14 a.

Alignment Fiducial Monitoring for Parking Operations

FIG. 14A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door tooutput an indication of a relative location of the aircraft to analignment fiducial indicating a parking location for the aircraft. FIG.14B illustrates aircraft 10 with aircraft skin 12, including doors 14 a,14 b, and 14 c having cameras 16 a (with field of view F_(a)), 16 b(with field of view F_(b)), and 16 c (with field of view F_(c)) disposedtherein, and alignment fiducials AF₁, AF₂, AF₃. FIGS. 14A-14B will bediscussed together. For purposes of clarity and ease of discussion, theexample operations of FIG. 14A are described below within the context ofcamera 16 a (FIGS. 1-6 and 14B) disposed within aircraft door 14 a(FIGS. 1-4 and 14B) and operatively connected to display device 20(FIGS. 2 and 5 ).

In certain examples, the operations of the example of FIG. 14A can beexecuted in response to receiving an indication that aircraft 10 is in ataxiing phase, such as an input from aircraft avionics 34 indicatingthat a phase of the aircraft operation indicates that aircraft 10 istaxiing. As such, the example operations, in certain examples, can beperformed only when the phase of aircraft operation indicates thattaxiing and alignment via an alignment fiducial indicating a parkinglocation are expected.

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 126). For example, processor 36 can receiveimage data captured by camera 16 a having a field of view that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 128). For instance, processor 36 canoutput the captured image data that is received from camera 16 a fordisplay at display device 20 that is disposed within an interior ofaircraft 10.

A region within the captured image data corresponding to an alignmentfiducial, such as alignment fiducial AF₁, indicating a parking locationfor the aircraft is identified (Step 130). The alignment fiducial caninclude, for example, intersecting orthogonal lines (as illustrated byalignment fiducials AF₁, AF₂, AF₃ in FIG. 14B) indicating an alignmentorientation and location for parking an aircraft for, e.g., dockingoperations. Processor 36 can analyze the captured image data byutilizing an object detection algorithm, such as the SSD algorithm forobject detection, the YOLO object recognition algorithm, or otherreal-time object detection algorithm to identify a region of thecaptured image data corresponding to the alignment fiducial. Forinstance, processor 36 can utilize the YOLO algorithm that has beentrained via image data of the alignment fiducial to recognize thealignment fiducial as an object, though in other examples, a differentobject detection algorithm can be used. Processor 36, executing, e.g.,the YOLO algorithm, can identify the region within the captured imagedata that corresponds to the alignment fiducial as the region of thebounding box produced by the YOLO algorithm that encloses the identifiedalignment fiducial object within the captured image data.

A relative location of the aircraft to the alignment fiducial isdetermined based on the region of the captured image data correspondingto the alignment fiducial (Step 132). The relative location can includea physical distance from at least a portion of the alignment fiducial aswell as relative orientation of the aircraft with respect to thealignment fiducial. Processor 36 can analyze the image data within theregion corresponding to the captured image data to extract relativelocation and alignment features from the image data within theidentified region corresponding to the captured image data. Relativelocation and alignment features can include, for example, a size of theregion within the captured image data corresponding to the alignmentfiducial (indicating, e.g., distance to the alignment fiducial), and/oran angle of intersection of the intersecting lines of the alignmentfiducial (indicating, e.g., a skewed alignment).

Processor 36 can utilize a relative location and alignment model, suchas a linear regression model or other model, to determine an extent towhich the region within the captured image data corresponding to thealignment fiducial correlates to baseline image data of the alignmentfiducial when the aircraft is aligned at a parking location indicated bythe alignment fiducial. For example, the relative location and alignmentmodel can be developed and trained using machine learning or othertechniques to produce an indication of an extent by which the extractedlocation and alignment features correlate to location and alignmentfeatures extracted from baseline (or training) image data captured bycamera 16 a while aircraft 10 is aligned at the parking locationindicated by the fiducial. Processor 36 can extract the location andalignment features from the captured image data and can utilize thetrained location and alignment model to produce an output that indicatesan extent to which the location and alignment features extracted fromthe captured image data received from camera 16 a correlate withalignment features extracted from the baseline image data, such as anormalized value that indicates the extent of correlation.

An indication of the relative location of the aircraft to the alignmentfiducial is output (Step 134). For example, processor 36 can output anindication of the relative location for display at display device 20 orother display device (e.g., a display device within the cockpit of theaircraft). In certain examples, the processor 36 can output theindication of the alignment state as a graphical overlay of targetlocation and alignment features over a graphical indication of theidentified location and alignment features. The indication of thealignment state can additionally or alternatively include instructionsto the cockpit regarding any necessary corrections to achieve analignment state. Additionally or alternatively, camera 16 a andprocessor 36 can form part of a system which can facilitate automatedalignment processes by leveraging camera recognition of alignmentfiducials such as ground surface markings and markings on the fixedstructure of a terminal. Camera 16 a and processor 36 can additionallyor alternatively coordinate with other systems within or external toaircraft 10 to facilitate alignment processes.

Accordingly, processor 36 can analyze captured image data received fromcamera 16 a to produce an indication of relative location and alignmentof aircraft 10 and an alignment fiducial that indicates a parkinglocation for aircraft 10, thereby assisting the flight crew or otherpersonnel in taxiing and parking operations of aircraft 10.Additionally, the use of a door camera such as camera 16 a can to allowfor effective, dynamic, and adaptive docking processes. This canadditionally increase the effectiveness and confidence of crew members.

Runway Collision Monitoring

FIG. 15 is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning of anticipated collision based on an identified regionwithin the captured image data that corresponds to an object separatefrom the aircraft at a region within the captured image data thatcorresponds to a runway intersection. For purposes of clarity and easeof discussion, the example operations of FIG. 15 are described belowwithin the context of camera 16 a (FIGS. 1-6 ) disposed within aircraftdoor 14 a (FIGS. 1-4 ) and operatively connected to display device 20(FIGS. 2 and 5 ).

In certain examples, the operations of the example of FIG. 15 can beexecuted in response to receiving an indication that aircraft 10 is in ataxiing to takeoff, takeoff, or taxiing to terminal phase of flight,such as an input from aircraft avionics 34 indicating that a phase offlight of the aircraft operation indicates that aircraft 10 is taxiingto takeoff, in takeoff, or taxiing to terminal. As such, the exampleoperations, in certain examples, can be performed only when the phase ofaircraft operation indicates that an encounter with a runwayintersection is expected.

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 136). For example, processor 36 can receiveimage data captured by camera 16 a having a field of view that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 138). For instance, processor 36 canoutput the captured image data that is received from camera 16 a fordisplay at display device 20 that is disposed within an interior ofaircraft 10.

A region within the captured image data that corresponds to a runwayintersection is identified (Step 140). For example, camera 16 a can bedisposed within a door of aircraft 10 such that the field of view ofcamera 16 a is oriented to capture image data corresponding to a runwayintersection ahead of aircraft 10. Processor 36 can analyze the capturedimage data to identify the captured image data that corresponds to arunway intersection. For instance, processor 36 can analyze the capturedimage data by utilizing an object detection algorithm, such as the SSDalgorithm for object detection, the YOLO object recognition algorithm,or other real-time object detection algorithm to identify a region ofthe captured image data that corresponds to a runway intersection. Forexample, processor 36 can utilize the YOLO algorithm that has beentrained via image data of runway intersections to recognize the runwayintersection as an object, though in other examples, a different objectdetection algorithm can be used. Processor 36 executing, e.g. the YOLOalgorithm, can identify the region within the captured image data thatcorresponds to the runway intersection as the region of the bounding boxproduced by the YOLO algorithm that encloses the identified runwayintersection object within the captured image data.

A motion vector of the aircraft relative to the region within thecaptured image data that corresponds to the runway intersection isdetermined (Step 142). For instance, processor 36 can determine themotion vector associated with the region within the captured image datareceived from camera 16 a that corresponds to the runway intersectionusing multiple frames of image data received from camera 16 a. In otherexamples, processor 36 can determine the trajectory of the identifiedrunway intersection based on a location of the identified runwayintersection within the captured image data and a motion vector of theidentified runway intersection determined based on, e.g. an Optical Flowor other motion tracking algorithm.

A region within the captured image data that corresponds to an objectseparate from the aircraft is identified (Step 144). For example,processor 36 can analyze the captured image data by utilizing an objectdetection algorithm, such as the SSD algorithm for object detection, theYOLO object recognition algorithm, or other real-time object detectionalgorithm to identify a region within the captured image datacorresponding to an object separate from aircraft 10. For instance,processor 36 can utilize the YOLO object recognition algorithm toidentify the region within the captured image data corresponding to anobject as the region within the captured image data that is output bythe YOLO algorithm as a bounding box around an identified object, thoughother object detection algorithms are possible. For instance, the YOLOcan be trained using baseline image data of objects to recognize any oneor more of a plurality of objects. Candidate objects can include, e.g.aircraft of various type and size, and/or vehicles of various type andsize. Processor 36, executing, a YOLO algorithm for example, candetermine a region of the captured image data corresponding to an objectthat is separate from the aircraft as a region of a bounding boxsurrounding an object that is produced by the YOLO algorithm.

A motion vector of the region that corresponds to the object relative tothe region within the captured image data that corresponds to the engineinlet is determined (Step 146). For instance, processor 36 can determinethe trajectory of the identified object based on a relative locationwithin the captured image data of the object within successive frames ofthe captured image data. In other examples, processor 36 can determinethe trajectory of the identified object based on a location of theidentified object within the captured image data and a motion vector ofthe identified object determined based on, e.g., an Optical Flow orother motion tracking algorithm.

An anticipated collision between aircraft 10 and the object separatefrom the aircraft at the identified runway intersection is identified(Step 148). For example, processor 36 can utilize the motion vectorcorresponding to the aircraft relative to the identified region thatcorresponds to the runway intersection and the motion vectorcorresponding to the object separate from the aircraft relative to theidentified region that corresponds to the runway intersection todetermine whether aircraft 10 and the object separate from the aircraftare anticipated to intersect at the location corresponding to the runwayintersection.

An incursion warning indicating an anticipated future collision isproduced (Step 150). For instance, processor 36 can output a visualalert for display at display device 20 and/or separate display devicewithin aircraft 10 (e.g., an EFIS display). In certain examples, such aswhen display device 20 includes a speaker device, the processor 36 cancause display 20 (or other audio output device) to produce an audiblealarm corresponding to the anticipated future collision. In someexamples, processor 36 can output an alert notification (e.g., a statusor other notification) to aircraft avionics or other aircraft systemsvia an aircraft communication data bus.

Accordingly, processor 36 that is operatively connected with camera 16 acan analyze captured image data received from camera 16 a to identify ananticipated incursion collision between the aircraft and an objectseparate from the aircraft. Processor 36 can output a warning of theanticipated collision, thereby alerting the pilots or other flight crewand increasing safety of operation of aircraft 10.

Evacuation Slide Deployment Monitoring

FIG. 16A is a flow chart illustrating example operations for utilizingcaptured image data from a camera disposed within an aircraft door toproduce a warning in response to determining that a region within thecaptured image data that corresponds to an evacuation slide deploymentpath is obstructed. FIG. 16B illustrates aircraft 10 with aircraft skin12, including doors 14 a, 14 b, and 14 c having cameras 16 a (with fieldof view F_(a)), 16 b (with field of view F_(b)), and 16 c (with field ofview F_(c)) disposed therein, evacuation slide deployment path 25, andobject O which is separate from aircraft 10. FIG. 16C shows evacuationslide deployment path 25 in a series of successful or failed deploymentoutcomes. For purposes of clarity and ease of discussion, the exampleoperations of FIG. 16A are described below within the context of camera16 a (FIGS. 1-6 ) disposed within aircraft door 14 a (FIGS. 1-4 ) andoperatively connected to display device 20 (FIGS. 2 and 5 ).

As described in more detail below, processor 36 can analyze the imagedata captured by camera 16 a in a variety of ways to determine that theevacuation slide deployment path 25 will generate a failed deploymentoutcome. This can include, for example, determining whether an objectseparate from aircraft 10 (such as object O, shown in FIG. 16B) willobstruct the evacuation slide deployment path. Processor 36 can furtherbe configured to produce a warning associated with the evacuation slidedeployment path in response to determining that the evacuation slidedeployment path will generate a failed deployment outcome (as describedbelow in reference to FIGS. 16B-16C).

In certain examples, the operations of the example of FIG. 16 can beexecuted in response to receiving an indication that aircraft 10 is inan emergency evacuation phase, such as input from aircraft avionics 24indicating that the phase of the aircraft operations indicates thatevacuation of the aircraft is imminent. As such, the example operations,in certain examples, can be performed only when the phase of aircraftoperation indicates that emergency slide deployment is expected.

Image data captured by a camera that is disposed within an aircraft doorand which has a field of view toward an external environment of theaircraft is received (Step 152). For example, processor 36 can receiveimage data captured by camera 16 a having a field of view that isoriented toward an exterior of aircraft 10. The captured image data isoutput for display at a display device that is disposed within aninterior of the aircraft (Step 154). For instance, processor 36 canoutput the captured image data that is received from camera 16 a fordisplay at display device 20 that is disposed within an interior ofaircraft 10.

A region within the captured image data corresponding to an evacuationslide deployment path is identified (Step 156). For example, camera 16 acan be disposed within a door of aircraft 10 such that the field of viewof camera 16 a is oriented to capture image data corresponding toevacuation slide deployment path 25. Processor 36 can analyze thecaptured image data to identify a region within the captured image datathat corresponds to evacuation slide deployment path 25. For instance,processor 36 can analyze the captured image data by identifying pixelswithin the captured image data that correspond to an expected locationof evacuation slide deployment path 25 within the captured image databased on the camera's fixed location relative to aircraft 10 and theevacuation slide deployment path. Processor 36 can identify the pixelswithin the captured image data that correspond to the location ofevacuation slide deployment path 25 as the region corresponding toevacuation slide deployment path 25. In other examples, other pathdetection algorithms can be used.

A region within the captured image data that corresponds to an objectseparate from the aircraft is identified (Step 158). For example,processor 36 can analyze the captured image data by utilizing an objectdetection algorithm, such as the SSD algorithm for object detection, theYOLO object recognition algorithm, or other real-time object detectionalgorithm to identify a region of the captured image data thatcorresponds to object O (shown in FIG. 16B) separate from the aircraftin the captured image data. For instance, processor 36 can utilize theYOLO object recognition algorithm to identify the region within thecaptured image data corresponding to object O as the region within thecaptured image data that is output by the YOLO algorithm as a boundingbox around an identified object separate from the aircraft, though otherobject detection algorithms are possible. For instance, the YOLOalgorithm can be trained using baseline image data of objects separatefrom the aircraft to recognize any one or more of a plurality ofobjects. Candidate objects can include, e.g. vegetation of various typesand sizes, and/or rocks of various types or sizes, and/or a groundsurface. Processor 36, executing, a YOLO algorithm for example, candetermine a region of the captured image data corresponding to an objectthat is separate from the aircraft as a region of a bounding boxsurrounding an object that is produced by the YOLO algorithm.

In some examples, candidate objects can include topological variationsof the ground surface. This can occur when, for example, aircraft 10 islocated on an incline or another surface which is not level and/oruniform, or when one set of landing gear for aircraft 10 fails. FIG. 16Cshows three potential locations (L₁, L₂, L₃) for evacuation slidedeployment path 25 relative to ground surface 27. As shown in FIG. 16C,evacuation slide deployment path 25 can be aligned with ground surface27 (location L₂, showing a successful landing gear deployment outcome)relative to ground surface 27. Evacuation slide deployment path 25 canalso be located above ground surface 27 (location L₁, a failed landinggear deployment outcome due to the collapse or other failure of the leftmain landing gear) or below ground surface 27 (location L₃, a failedlanding gear deployment outcome due to the collapse or other failure ofthe right main landing gear). The relative location of evacuation slidedeployment path 25 to ground surface 27 can vary based on the surface,potential obstructions, aircraft orientation, or other factors. Thefield of view F_(a) of camera 16 a can determine if obstructions arepresent within evacuation slide deployment path 25 even if the left orright main landing gear fail during landing.

It is determined whether the object separate from the aircraft obstructsevacuation slide deployment path 25 (Step 159). For example, processor36 can determine the intersection of the region of the captured imageddata corresponding to evacuation slide deployment path 25 and the regionof the captured image data corresponding to an object separate from theaircraft, such as object O. For instance, processor 36 can determine apixel location of the object separate from the aircraft within thecaptured image data. The pixel location can be converted to a physicallocation based on the known field of view of camera 16 a relative to alocation of camera 16 a on aircraft 10. Processor 36 can determine,based on the physical location of object O and the region of thecaptured image data corresponding to evacuation slide deployment path25, that the object separate from the aircraft and evacuation slidedeployment path 25 intersect. Processor 36 can determine that object Ois obstructing evacuation slide deployment path 25 in response toidentifying the intersection.

A warning associated with the evacuation slide deployment is produced inresponse to determining that evacuation slide deployment path 25 isobstructed (Step 160). For example, processor 36 can output a visualalert for display at display device 20 and/or a separate display devicewithin aircraft 10 (e.g., an EFIS display). In certain examples, such aswhen display device 20 includes a speaker device, processor 36 can causedisplay 20 (or other audio output device) to produce an audible alarmcorresponding to the warning associated with the obstruction ofevacuation slide deployment path 25. In some examples, processor 36 canoutput an alert notification (e.g., a status or other notification) toaircraft avionics or other aircraft systems via an aircraftcommunication data bus. A failed evacuation slide deployment outcome dueto an obstacle such as object O intersecting with evacuation slidedeployment path 25 can also trigger a warning associated with theevacuation slide deployment. This warning can additionally oralternatively take the form of a green/red light system whichcommunicates to crew members whether aircraft 10 should continue taxiingto a different location (a “red light” scenario in which the evacuationslide should not be deployed) or if the present location of aircraft 10is suitable for deployment of the evacuation slide (a “green light”scenario). This warning can optionally include an alert that an aircraftdoor is being opened before a suitable location for the evacuation slidehas been reached and/or a communication regarding directions to thenearest deployed evacuation slide on the aircraft.

Accordingly, techniques of this disclosure can enable processor 36 toanalyze captured image data received from camera 16 a to produce awarning in response to determining that an evacuation slide deploymentpath is obstructed, thereby alerting the pilots or other flight crew andincreasing safety of operation of aircraft 10. Captured image data fromany of cameras 16 a, 16 b, 16 c, 16 d can be aggregated to gatheradditional information about the external environment of aircraft 10.Additionally, camera 16 a and processor 36 can form part of a systemwhich can automate the evacuation slide deployment process and/or ensurethat the evacuation slide is not deployed before a suitable location forthe evacuation slide has been reached. This can help to preventdeployment of the evacuation slide when it is unnecessary.

Aggregation of Image Data Captured From Multiple Cameras

FIG. 17 is a flow chart illustrating example operations for utilizingcaptured image data from each of a plurality of cameras disposed withinaircraft doors to output the aggregated image data for display. Forpurposes of clarity and ease of discussion, the example operations ofFIG. 17 are described below within the context of cameras 16 a, 16 b, 16c, and 16 d (FIG. 6 ) disposed within aircraft doors 14 a, 14 b, 14 c,and 14 d (FIG. 6 ).

Image data captured by each camera of the plurality of cameras disposedwithin aircraft doors and which has a unique field of view with respectto each other camera of the plurality of cameras toward an externalenvironment of the aircraft is received (Step 162). For example,processor 36 can receive image data captured by cameras 16 a, 16 b, 16c, and 16 d, each having a unique field of view among the group ofcameras 16 a, 16 b, 16 c, and 16 d, and having fields of view orientedtoward an exterior of aircraft 10.

Image data captured by each camera of the plurality of cameras isaggregated such that image data from overlapping fields of view ofcameras 16 a, 16 b, 16 c, and 16 d is presented only once in theaggregated image data (Step 164). For instance, processor 36 canaggregate the captured image data by utilizing an image stitchingalgorithm, such as Keypoint, Registration, or other real-time imagestitching algorithms to aggregate the captured image data such thatimage data corresponding to overlapping fields of view is presented onlyonce in the aggregated image data.

As an example, processor 36 can utilize a Keypoint algorithm to detectand describe key features in image data captured by each camera of theplurality of cameras. For instance, the Keypoint algorithm can useHarris Corner Detection, Scale Invariant Feature Transform (SIFT),Speed-Up Robust Features (SURF) algorithms, or other feature detectionand description algorithms to detect and describe key features in imagedata captured by each of the plurality of cameras. The Keypointalgorithm matches key features detected in multiple image data capturedby each of the plurality of cameras, e.g., using the Euclidean distancebetween each key feature of each image data captured by each camera ofthe plurality of cameras. Processor 36 can transform and warp each imagedata from the captured image data from each of a plurality of cameras toalign each of the matched key features, e.g., using the Random SampleConsensus algorithm (RANSAC). Processor 36 stitches each transformed andwarped image data from each camera of the plurality of cameras whileimage data from overlapping fields of view is presented only once in theaggregate image data.

Aggregated image data is output for display (Step 166). For instance,processor 36 can output the aggregated image data with image data fromoverlapping fields of view presented only once at display device 20 orother display device.

Accordingly, processor 36 can analyze captured image data received fromeach camera of the cameras 16 a, 16 b, 16 c, and 16 d to produceaggregated image data with image data from overlapping fields of viewpresented only once, thereby providing the pilots or other flight crewan understanding of the external environment of aircraft 10, increasingthe safety of operations of aircraft 10. Processor 36 can furtheranalyze the aggregated image data by, for example, performing thefunctions described in reference to FIGS. 7A-16C—in particular,identifying a region corresponding to an alignment fiducial indicating aparking location for the aircraft, determining a relative location ofthe aircraft to the alignment fiducial based on the region of thecaptured image data corresponding to the alignment fiducial indicatingthe parking location, and outputting an indication of the relativelocation of the aircraft to the alignment fiducial.

At least one of cameras 16 a, 16 b, 16 c, 16 d can be disposed within anaircraft door (such as doors 14 a, 14 b, 14 c, 14 d). To monitor thedocking fiducials, at least one camera of cameras 16 a, 16 b, 16 c, 16 dcan be oriented such that the ground surface is within the at least onecamera's field of view. The location of cameras 16 a, 16 b, 16 c, 16 dcan be varied to provide at least one field of view which includes theregion to be monitored (here, the ground surface containing the dockingfiducials).

Discussion of Potential Embodiments

The following are non-exclusive descriptions of possible embodiments ofthe present invention.

A system for monitoring an external environment of an aircraft includesan aircraft door, a camera, a display device, and a processor. Thecamera has a field of view toward the external environment of theaircraft and is disposed within an aircraft door such that a groundsurface is within the field of view of the camera during taxiing of theaircraft. The display device is disposed within an interior of theaircraft. The processor is operatively coupled to the camera and displaydevice. The processor receives image data captured by the camera that isrepresentative of the external environment of the aircraft and outputsthe captured image data for display at the display device. The processoranalyzes the captured image data for docking guidance by: identifying,within the captured image data, a region on the ground surfacecorresponding to an alignment fiducial indicating a parking location forthe aircraft, determining, based on the region of the captured imagedata corresponding to the alignment fiducial indicating the parkinglocation, a relative location of the aircraft with respect to thealignment fiducial, and outputting an indication of the relativelocation of the aircraft with respect to the alignment fiducial.

The system of the preceding paragraph can optionally include,additionally and/or alternatively, any one or more of the followingfeatures, configurations, and/or additional components:

A system for monitoring an external environment of an aircraft,according to an exemplary embodiment of this disclosure, among otherpossible things includes an aircraft door, a camera, a display device,and a processor. The camera has a field of view toward the externalenvironment of the aircraft and is disposed within an aircraft door suchthat a ground surface is within the field of view of the camera duringtaxiing of the aircraft. The display device is disposed within aninterior of the aircraft. The processor is operatively coupled to thecamera and display device. The processor receives image data captured bythe camera that is representative of the external environment of theaircraft and outputs the captured image data for display at the displaydevice. The processor analyzes the captured image data for dockingguidance by: identifying, within the captured image data, a region onthe ground surface corresponding to an alignment fiducial indicating aparking location for the aircraft, determining, based on the region ofthe captured image data corresponding to the alignment fiducialindicating the parking location, a relative location of the aircraftwith respect to the alignment fiducial, and outputting an indication ofthe relative location of the aircraft with respect to the alignmentfiducial.

A further embodiment of the foregoing system, wherein the processor isoperatively coupled to the aircraft to receive an indication that theaircraft is in a taxiing phase.

A further embodiment of any of the foregoing systems, wherein theprocessor is operatively coupled to an avionics system of the aircraft.

A further embodiment of any of the foregoing systems, wherein theprocessor is operatively coupled to the camera to identify a regionwithin the captured image data that corresponds to an alignment fiducialby utilizing an object detection algorithm.

A further embodiment of any of the foregoing systems, wherein therelative location of the aircraft comprises at least one of a size ofthe region corresponding to the alignment fiducial and a skew of thealignment fiducial.

A further embodiment of any of the foregoing systems, wherein thealignment fiducial comprises a plurality of intersecting line segmentsand the skew of the alignment fiducial comprises an angle ofintersection between the intersecting line segments.

A method of monitoring an external environment of an aircraft includesreceiving, with a processor, image data captured by a camera disposedwithin an aircraft door of the aircraft such that a ground surface iswithin a field of view of the camera during taxiing of the aircraft. Theprocessor analyzes the captured image data for docking guidance by:identifying, within the captured image data, a region on the groundsurface corresponding to an alignment fiducial indicating a parkinglocation for the aircraft, determining, based on the region of thecaptured image data corresponding to the alignment fiducial indicatingthe parking location, a relative location of the aircraft with respectto the alignment fiducial, and outputting an indication of the relativelocation of the aircraft with respect to the alignment fiducial. Thecaptured image data is output for display at a display device disposedwithin an interior of the aircraft.

The method of the preceding paragraph can optionally include,additionally and/or alternatively, any one or more of the followingfeatures, configurations, and/or additional components:

A method of monitoring an external environment of an aircraft, accordingto an exemplary embodiment of this disclosure, among other possiblethings includes receiving, with a processor, image data captured by acamera disposed within an aircraft door of the aircraft such that aground surface is within a field of view of the camera during taxiing ofthe aircraft. The processor analyzes the captured image data for dockingguidance by: identifying, within the captured image data, a region onthe ground surface corresponding to an alignment fiducial indicating aparking location for the aircraft, determining, based on the region ofthe captured image data corresponding to the alignment fiducialindicating the parking location, a relative location of the aircraftwith respect to the alignment fiducial, and outputting an indication ofthe relative location of the aircraft with respect to the alignmentfiducial. The captured image data is output for display at a displaydevice disposed within an interior of the aircraft.

A further embodiment of the foregoing method, wherein identifying,within the captured image data, a region corresponding to an alignmentfiducial comprises utilizing an object detection algorithm.

A further embodiment of any of the foregoing methods, whereindetermining, based on the region of the captured image datacorresponding to the alignment fiducial indicating the parking location,a relative location of the aircraft with respect to the alignmentfiducial comprises utilizing a relative location and alignment model todetermine an extent to which the region within the captured image datacorresponding to the alignment fiducial correlates to baseline imagedata of the alignment fiducial when the aircraft is aligned at a parkinglocation indicated by the alignment fiducial.

A further embodiment of any of the foregoing methods, wherein utilizingthe relative location and alignment model to determine the extent towhich the region within the captured image data corresponding to thealignment fiducial correlates to baseline image data of the alignmentfiducial comprises extracting location and alignment features from thecaptured image data, and utilizing the location and alignment model toproduce an output that indicates the extent to which the location andalignment features extracted from the captured image data correlate withalignment features extracted from the baseline image data.

A system of monitoring an external environment of the aircraft includesa plurality of aircraft doors, a plurality of cameras, a display device,and a processor. At least one of the plurality of cameras are disposedwithin one of the aircraft doors and each of the plurality of camerashave a field of view that is unique among the plurality of cameras. Aground surface is within the field of view of at least one camera duringtaxiing of the aircraft. The display device is disposed within aninterior of the aircraft. The processor is operatively coupled to thecamera and display device to: receive, from each respective camera ofthe plurality of cameras, image data captured by the respective camerathat is representative of the external environment of the aircraftwithin the field of view of the respective camera, aggregate thecaptured image data received from each camera of the plurality ofcameras to produce aggregated image data representative of the externalenvironment of the aircraft, wherein image data from overlapping fieldsof view of the plurality of cameras is presented only once in theaggregated image data, and output the aggregated image data for displayat the display device. The processor analyzes the aggregated image datafor docking guidance by: identifying, within the aggregated image data,a region on the ground surface corresponding to an alignment fiducialindicating a parking location for the aircraft, determining, based onthe region of the aggregated image data corresponding to the alignmentfiducial indicating the parking location, a relative location of theaircraft with respect to the alignment fiducial, and outputting anindication of the relative location of the aircraft with respect to thealignment fiducial.

The system of the preceding paragraph can optionally include,additionally and/or alternatively, any one or more of the followingfeatures, configurations, and/or additional components:

A system for monitoring an external environment of an aircraft,according to an exemplary embodiment of this disclosure, among otherpossible things includes a plurality of aircraft doors, a plurality ofcameras, a display device, and a processor. At least one of theplurality of cameras are disposed within one of the aircraft doors andeach of the plurality of cameras have a field of view that is uniqueamong the plurality of cameras. A ground surface is within the field ofview of at least one camera during taxiing of the aircraft. The displaydevice is disposed within an interior of the aircraft. The processor isoperatively coupled to the camera and display device to: receive, fromeach respective camera of the plurality of cameras, image data capturedby the respective camera that is representative of the externalenvironment of the aircraft within the field of view of the respectivecamera, aggregate the captured image data received from each camera ofthe plurality of cameras to produce aggregated image data representativeof the external environment of the aircraft, wherein image data fromoverlapping fields of view of the plurality of cameras is presented onlyonce in the aggregated image data, and output the aggregated image datafor display at the display device. The processor analyzes the aggregatedimage data for docking guidance by: identifying, within the aggregatedimage data, a region on the ground surface corresponding to an alignmentfiducial indicating a parking location for the aircraft, determining,based on the region of the aggregated image data corresponding to thealignment fiducial indicating the parking location, a relative locationof the aircraft with respect to the alignment fiducial, and outputtingan indication of the relative location of the aircraft with respect tothe alignment fiducial.

While the invention has been described with reference to an exemplaryembodiment(s), it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the scope of the invention. Inaddition, many modifications may be made to adapt a particular situationor material to the teachings of the invention without departing from theessential scope thereof. Therefore, it is intended that the inventionnot be limited to the particular embodiment(s) disclosed, but that theinvention will include all embodiments falling within the scope of theappended claims.

1. A system for monitoring an external environment of an aircraft, thesystem comprising: an aircraft door; a camera with a field of viewtoward the external environment of the aircraft and disposed within theaircraft door such that a ground surface is within the field of view ofthe camera during taxiing of the aircraft; a display device disposedwithin an interior of the aircraft; and a processor operatively coupledto the camera and to the display device to: receive image data capturedby the camera that is representative of the external environment of theaircraft; output the captured image data for display at the displaydevice; and analyze the captured image data for docking guidance by:identifying, within the captured image data, a region on the groundsurface corresponding to an alignment fiducial indicating a parkinglocation for the aircraft; determining, based on the region of thecaptured image data corresponding to the alignment fiducial indicatingthe parking location, a relative location of the aircraft with respectto the alignment fiducial; and outputting an indication of the relativelocation of the aircraft to the alignment fiducial.
 2. The system ofclaim 1, wherein the processor is operatively coupled to the aircraft toreceive an indication that the aircraft is in a taxiing phase.
 3. Thesystem of claim 2, wherein the processor is operatively coupled to anavionics system of the aircraft.
 4. The system of claim 1, wherein theprocessor is operatively coupled to the camera to identify a regionwithin the captured image data that corresponds to an alignment fiducialby utilizing an object detection algorithm.
 5. The system of claim 1,wherein the relative location of the aircraft comprises at least one ofa size of the region corresponding to the alignment fiducial and a skewof the alignment fiducial.
 6. The system of claim 5, wherein thealignment fiducial comprises a plurality of intersecting line segmentsand the skew of the alignment fiducial comprises an angle ofintersection between the intersecting line segments.
 7. A method ofmonitoring an external environment of an aircraft, the methodcomprising: receiving, with a processor, image data captured by a cameradisposed within an aircraft door of the aircraft such that a groundsurface is within a field of view of the camera during taxiing of theaircraft; analyzing, with the processor, the captured image data fordocking guidance by: identifying, within the captured image data, aregion on the ground surface corresponding to an alignment fiducialindicating a parking location for the aircraft; determining, based onthe region of the captured image data corresponding to the alignmentfiducial indicating the parking location, a relative location of theaircraft with respect to the alignment fiducial; and outputting anindication of the relative location of the aircraft to the alignmentfiducial; and outputting the captured image data for display at adisplay device disposed within an interior of the aircraft.
 8. Themethod of claim 7, wherein identifying, within the captured image data,a region corresponding to an alignment fiducial comprises utilizing anobject detection algorithm.
 9. The method of claim 7, whereindetermining, based on the region of the captured image datacorresponding to the alignment fiducial indicating the parking location,a relative location of the aircraft with respect to the alignmentfiducial comprises utilizing a relative location and alignment model todetermine an extent to which the region within the captured image datacorresponding to the alignment fiducial correlates to baseline imagedata of the alignment fiducial when the aircraft is aligned at a parkinglocation indicated by the alignment fiducial.
 10. The method of claim 9,wherein utilizing the relative location and alignment model to determinethe extent to which the region within the captured image datacorresponding to the alignment fiducial correlates to baseline imagedata of the alignment fiducial comprises: extracting location andalignment features from the captured image data; and utilizing thelocation and alignment model to produce an output that indicates theextent to which the location and alignment features extracted from thecaptured image data correlate with alignment features extracted from thebaseline image data.
 11. A system for monitoring an external environmentof an aircraft, the system comprising: a plurality of aircraft doors; aplurality of cameras, at least one camera of the plurality of camerasbeing disposed within one of the plurality of aircraft doors and eachcamera of the plurality of cameras having a field of view that is uniqueamong the plurality of cameras, such that a ground surface is within thefield of view of at least one camera during taxiing of the aircraft; adisplay device disposed within an interior of the aircraft; and aprocessor operatively coupled to the camera and to the display deviceto: receive, from each respective camera of the plurality of cameras,image data captured by the respective camera that is representative ofthe external environment of the aircraft within the field of view of therespective camera; aggregate the captured image data received from eachcamera of the plurality of cameras to produce aggregated image datarepresentative of the external environment of the aircraft, whereinimage data from overlapping fields of view of the plurality of camerasis presented only once in the aggregated image data; analyze theaggregated image data by: identifying, within the aggregated image data,a region on the ground surface corresponding to an alignment fiducialindicating a parking location for the aircraft; determining, based onthe region of the aggregated image data corresponding to the alignmentfiducial indicating the parking location, a relative location of theaircraft to the alignment fiducial; and outputting an indication of therelative location of the aircraft to the alignment fiducial; and outputthe aggregated image data for display at the display device.